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Abstract 


I'Jie  ever-decreasing  cost  of  computer  processors  has  created  a  great  interest  in  multiprocessor  com¬ 
puters.  However,  along  with  the  increased  power  that  this  parallelism  brings,  comes  increased 
complexity  in  programming. 

One  approach  to  les.scning  this  complexity  is  to  provide  the  programmer  with  general  purpose 
parallel  primitives  that  shield  him  from  the  structure  of  the  underlying  machine.  There  are  two 
contending  goafs  (hat  mu.st  be  sa(isfied  when  designing  primitives.  On  one  hand  we  would  like  to 
make  the  primitive  as  general  and  ab.stract  as  possible.  The  more  general  a  primitive  is,  the  more 
easily  it  can  be  used  as  a  building  block  for  creating  complex  algorithmic  constructs.  On  the  other 
hand,  we  want  to  avoid  niaking  the  primitive  so  general  that  it  becomes  inefficient. 

In  2'hc  Connection  Machine  [Mil85],  Hillis  suggests  the  beta  operation  as  a  parallel  primitive 
for  his  hy|iercnhe-ba.sed  machine.  We  shall  examine  the  beta  operation,  demonstrate  its  efficiency, 
generalize  it  in  st'veral  directions,  and  show  its  suitability  for  general  use. 
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Chapter  1 

Introduction 


The  ever-decreasing  cost  of  computer  processors  has  fueled  an  increased  interest  in  mnlti))rocessor 
networks  and  parallel  computation.  Much  work  has  been  performed  on  producing  eiricient  parallel 
versions  of  se(|uentiai  algoritlims.  These  jrarallel  algoritlirns  present  considerably  more  conrplex 
programming  reciuirements  than  their  associated  secinential  versions. 

A  standard  method  of  lessening  the  complexity  of  .secinential  programming  is  to  pio\idi'  the 
programmer  with  abstract  data  structures  and  operations  that  shield  him  from  the  structure'  of  the 
underlying  machine.  The  problem  of  creating  an  algorithm  for  a  particular  problem  i  lu'ii  divides 
into  two  simpler  problems,  namely,  creating  an  algorithm  in  terms  of  certain  data  structures  and 
implementing  the  data  structures  on  a  particular  machine. 

The  advantages  of  this  method  are  twofold.  The  details  of  how  data  are  to  be  moved  are  hidden 
in  the  implementation  of  the  data  structures.  Also,  once  an  algorithm  is  expressed  in  tc'rms  of  data 
structures  and  their  manipulation,  it  can  be  u.sed  on  any  machine. 

The  motivation  is  the  same  for  studying  parallel  primitives.  We  want,  to  shield  the  algorithm 
designer  from  details  of  the  underlying  network.  On  the  one  hand,  we  want  to  determine  the 
relationship  between  various  parallel  primitives  and  particular  networks.  On  tlie  other  baud,  we 
would  like  to  study  how  various  parallel  algoritlims  can  be  clecompo.sed  into  a  set  of  priinillvc 
operations.  The  parallel  primitive  operations  can  be  used  a.s  building  blocks  for  creating  complex 
algorithmic  constructs,  making  it  possible  to  reason  about  and  describe  algoiilhms  without  explicit 
reference  to  the  networks.  This  approach  has  been  explored  by  a  number  of  reseanilel^,  ([|[or8.')]. 
[RB.J88],  [Ilna85],[Ble87]). 

There  are  two  contending  goals  that  must  be  satisfied  when  designing  a  parallel  primitive.  It  is 
desirable  to  make  the  primitive  as  general  eis  possible,  for  the  more  general  it  is,  the  moie  readily 
it  can  be  employed  to  create  algorithms.  However,  increasing  generality  is  ofti'ii  accomi>anied  by  a 
loss  of  eiriciency.  At  some  point  the  primitive  becomes  biglily  general  and  very  inefficient.  It  is  of 
great  importance,  therefore,  to  find  an  optimal  balance. 

In  The  Conveciton  Machine  [IIil85],  Hillis  proposes  tlie  beta  opeiatinn  as  a  suitable  parallel 
primitive  for  his  liyperciibe-ba,sed  machine.  In  the  subsequent  chajiters,  we  shall  examine  tlu'  beta 
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oi'eration,  cleinoiisirat.e  its  efficiency,  generalize  it  in  several  directions  and  show  tliat  it  is  suitable 
for  gc'iieral  use. 


1.1  Thesis  Organization 

Id  Chapter  2.  we  /iresent  two  vziriant  formulations  of  the  l)eta  operation  and  introduce  some  notation. 

Id  Cliapler  3,  we  study  the  problem  of  implementing  the  beta  operation  on  several  multiprocessor 
networks,  namely,  the  hypercube,  mesh,  and  mesh-of-trees  graphs.  The  hypercube  implementation 
is  presenled  first  and  serves  as  a  paradigm  for  the  other  implementations.  AT-  lower  bounds  are 
pri'sented  for  circuits  implementing  the  beta  operation.  These  bounds  are  close  to  the  upper  bounds 
realized  in  the  mesh-of-trees  implementation. 

In  Chapter  4,  the  beta  operation  is  generalized  to  cover  all  possible  relationships  between  the 
number  of  proces.sois,  the  number  of  input  pairs  and  the  number  of  groups  represented  by  these 
jiivirs. 

In  Chapter  -5,  we  re-examine  the  hypercube  implementation  of  the  beta  operation  from  Chap- 
t('r  3  anil  consider  the  implementation  of  a  new  primitive  that  generalizes  the  beta  operation,  the 
multi-prefix  operation.  We  show  that  by  creating  a  circuit  that  stores  information  about  the  partial 
computations  that  arise  during  the  hypercube  implementation  of  the  beta  operation,  we  can  imple¬ 
ment  till'  nuilti-prerix  operation  with  no  increase  in  complexity.  We  can  use  this  implementation  as 
a  paradigm  for  implementing  the  multi-prefix  operation  efficiently  on  other  networks. 

In  Chapter  (i,  we  present  some  applications  of  the  beta  operation.  We  first  demonstrate  both 
the  efficiency  niul  the  generality  of  the  beta  operation  by  simulating  complex  beta  operations  using 
only  simpler  beta  operations,  with  only  a  constant  factor  increa.se  in  the  time  rec|uired.  We  next 
show  a  sample  application  of  one  of  the  results  established  in  Chapter  4. 

In  Chajiter  7,  we  review  the  results  presented  and  consider  open  questions. 


Chapter  2 

The  Beta  Operation 


2.1  Definition 

We  begin  by  pieseiif  ing  the  detiiiition  of  the  beta  operation.  As  an  instance  of  the  beta  operation, 
\v('  are  given  an  a.ssociative,  coiimiiitative  binary  function  F  and  N  pairs,  (ffo>  *'o),  •  •  •  •  (.</iV-i ,  t’/V-i)- 
I'lie  ((j's  .should  he  thought  of  as  f/roiip  ntiT>ihe7’s  and  tlie  Vj’s  as  data.  Let  us  denote  by  Bi,  tlie 
collection  of  (i(,  e)-pairs  with  </  =  i  (for  some  v).  We  shall  call  the  set  of  processors  holding  these 
pairs  tet  G  =  {/'|  |/t  |  >  0}.  We  .shall  assume  that  every  processor  has  a  unique  binary 

address  of  length  log  P  The  processors  of  proc{^B^  are  ordered  by  their  binary  addresses. 

For  a  two-argument  function  F  and  a  list  of  values  C  =  [co, . . . ,  Cm],  let  us  define  the  F -reduction 
of  ('  a.s  the  natural  reduction.  That  is 


•  f'/h]  =  Co; 

•  F/[co,...,  Cm]  —  T(F/[co,...,  e?n—  l],  Cm ) 

If  li  is  the  list  of  e-values  from  Bi,  then  the  beta  operation  computes  the  output  values  y,-  =  F//,- 
for  i  €  G  and  produces  pairs  (/.?/,) 

For  .simplicity,  we  shall  hereafter  refer  to  |G'|  as  S.  W'e  shall  call  a  beta  operation  with  N  input 
pairs,  P  proce.ssors,  and  S  output  groups,  ld{N,P.S).  If  the  values  of  N  and  P  are  understood, 
we  shall  call  the  operation  simply  I3{S).  We  shall  refer  to  two  variant  formulations  of  the  beta 
operation.  We  shall  term  them  fl\(S)  and  If  the  (>,  j/i)  pairs  end  up  stored,  sorted  liy  y-value, 

in  the  S  bigliest-addre.ss  (lowest -address)  processors,  we  shall  call  the  operation  (respectively. 

Pi  W('  sliall  call  tlie  operation  simply  pp.'^)  if  the  sign  of  the  superscript  is  irot  important.  The 

description  of  tlie  p-j{S)  operat  ion  is  identical  except  ilial  tlie  yPs  are  distributed  to  the  processors 
so  tliat  at  the  coriiplet ion,  all  the  meniliers  of  hold  y,-. 

Example  2.1.1  Let  P  bf  '+'  and  N  b(  .%  Let  the  (<;,  r)-paiT'.s  m  Hit  input  be: 

prncl:(f),())  jiror  2:  (2.1)  proc ',i:  A]  yrer  ■) .  (9,  5)  proc5:(2,b) 
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Performing  on  these  pairs  produces  the  output: 

procZ.  (2,9)  proc4:{3,4)  pror  5.' (9,  1 1) 


Example  2.1.2  Let  F  be  ‘-f  ’  and  N  be  5.  Let  (he  (g,  ii)-pa/rs  tn  the  input  be: 

procl;(9,6)  proc  2;  (2,4)  proc3;(3,4)  proc4.(9,5)  proc5:(2,5} 

Performing  3^(3)  on  these  pairs  produces  the  output: 

proc  1;  (9, 11)  pi'oc2;(2,9)  proc  3;  (2,4)  p70c  4;  (9. 1 1)  proc  5;  (2,9) 

We  shall  use  the  word  model  of  computation  in  our  multiprocessor  networks.  In  this  model,  every 
processor  has  a  local  memory  with  words  of  length  O(logA^);  the  standard  set  of  ALU  operations 
can  be  performed  in  constant  time  on  the.se  words.  In  our  model  of  the  hypercube,  each  processor 
sends  or  receiver  a  single  message  in  each  time  step. 

2.2  Notation 

For  many  of  the  algorithms  presented  below,  we  shall  need  a  simple  way  to  refer  to  sets  of  processors. 
For  convenience  in  reference,  we  shall  view  ]>rocessor  addresses  as  being  composed  of  several  Rehls, 
(e.g.,  UkVkWki'k)-  shall  u.se  the  convention  tliat  the  capitalized  versions  of  the  field  names  (e.g., 
Vk  for  field  la)  will  represent  a  value  for  that  field,  that  is,  a  fixed  string  of  the  appropriate  length. 
We  shall  also  use  the  convention  that  a  value  with  an  underscore  beneath  it  (e.g.,  V^)  will  represtint 
the  .set  of  all  po.ssible  strings  of  the  appropriate  length.  Thus,  Ui-VkWkXk  represents  the  2l^‘l 
processor  addresses  fh-VtW'jtA't  for  0  <  A't  <  2'^*!,  and  Uk  in  field  iik,  Vk  in  field  Vk,  and  l-V).  in 
field  Wk. 

For  simplicity,  we  shall  assume  that  S  is  a  jiower  of  2.  It  is  true  that  2-’~'  <  S  <  2-'  for  .some  j. 
If  vve  cissume  that  5  is  really  2C  the  algorithms  of  the  following  chapters  will  work  with  the  same 
asymptotic  time  complexities.  We  sliall  call  log 5,  s.  We  sliall  call  S  —  1,  9,  and  5  —  2,  eji-  Note 
that  the  binary  representation  of  the  former  is  a  string  of  s  I’s  and  the  binary  representation  of  the 
later  is  a  string  of  .s  -  1  I’s  followed  by  a  zero.  We  shall  use  the  convention  that  a  digit,  d,  with 
a  superscript,  k.  (e.g.,  9^')  will  be  shorthand  for  the  number  whose  binary  representation  is  a  string 
of  k  consecutive  d's. 

We  shall  call  the  operation  of  sorting  N  items  on  a  F-node  network,  SORT(iV,  P).  The  identity 
of  the  network  in  ([ucstion  will  be  understood  by  context.  We  shall  call  t  he  minimum  time  re<|uired 
to  perform  SORT(/V,F)  on  this  network,  '1'sokt(X ,  P). 


Chapter  3 


Implementing  the  Beta  Operation 


3.1  Overview 

111  t.)iis  chapter,  we  shall  consider  the  problem  of  implcMnonting  on  several  multipro¬ 
cessor  networks.^  In  Section  3. 2,  wo  present  an  efficient  hypermbe  iiii|)lementation  of  N,S). 

In  .Section  3.3,  we  show  how  0i'^(N,N,S)  can  be  implemented  on  a  mesli-of-trees  network,  using 
the  hypercubc  implementation  as  a  paradigm.  Additionally,  the  hyperciibe  implementation  can  be 
used  as  a  paradigm  for  implementing  N,S)  on  the  butterfly  and  shuITle-e,\'change  networks. 

Both  of  the  implementations  that  we  present  assume  that  the  value  of  .5’  is  known.  In  Section  3.4, 
we  show  that  the  same  time  bounds  can  be  achieved  without  knowing  S  beforehand.  Finally,  in 
.Section  3.5  we  present  an  AT-  lower  bound  that  is  only  a  few  factors  of  log  below  the  upper  bound 
presented  in  Section  3.3. 

3.2  Efficient  Hypercube  Implementation 

We  start  by  considering  the  efficient  hypercubc  implementation  of  N,S).  Let  W  =  2"  and 

5  =  2*.  We  shall  call  n/s,  D.  The  case  where  s  does  not  divide  n  evenly  can  be  treated  by  trivially 
modifying  the  algorithm  to  follow.  The  proces.sor  addresses  will  be  viewed  as  being  composed  of 
D  base-S  digits.  We  shall  view  these  base- .S’  addresses  as  being  composed  of  four  fields,  UkVkWkXt, 
where  k  is  the  number  of  the  step  we  are  performing.  The  lengths  of  fields  tir  and  Wk  are  functions 
of  k.  The  fields  are  composed  as  shown  in  Figure  3.1.  The  implementation  will  require  time 

O(log  N  -(-  /'sORTf'?,  'S')). 


3.2.1  The  General  Step 

In  each  step  of  this  algorithm,  we  shall  conceptually  lueak  the  hypeucube  into  a  number  of  smaller 
subcubes.  In  each  step  of  Phases  1  and  2,  we  shall  apply  the  thii'e  .s/rtc/f.s  shown  lielow  to  all  of  the 

'Much  of  the  material  of  this  chapter  wa.s  lii-s!  presented  in  ((Ml8b]. 
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Figure  3.1:  Composition  of  Hypercube  Addresses. 
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Figure  3.2:  Hierarcity  between  Phases,  Steps,  and  Stages. 


subcubes  in  parallel.  The  hierarchy  between  phases,  steps,  and  stages  is  illustrated  in  Figure  3.2. 

Sort  Stage.  R.ecall  that  B,  is  the  collection  of  (3,t>)-pairs  with  g  =  ».  Consider  a  subcube,  H.  For 
convenience,  we  shall  use  Bi(H)  to  refer  to  the  set  of  (ff,  v)-pairs  in  H  from  a  particular  Bi. 
We  shall  call  the  set  of  processors  holding  these  pairs,  proc(^Bi{H)^ .  We  sort  the  (g,  v)-pairs 
in  H  by  y-value.  The  pairs  of  each  Bi(H)  will  now  reside  in  consecutively  numbered  processors 
in  H . 

Reduce  Stage.  We  next  reduce  each  Bi{H)  to  a  single  (r/,  t>)-pair  by  applying  the  function  F  to 
the  associated  v-values. 

Compact  Stage.  Finally,  we  route  the  resulting  (</,  t;)-pairs,  of  which  there  are  no  more  than  5, 
into  the  liighest-numbered  processors  of  the  subcube. 

3.2.2  Auxiliary  Lemmas 

During  the  reduce  stage  of  Phase  1,  every  B,(//)  will  be  reduced  to  a  single  value  in  parallel  in 
each  subcube  H.  Before  presenting  the  liypercube  implementation  of  Pi'^{N,N,S),  we  shall  first 
establish  two  auxiliary  lemmas  that  show  that  this  reduction  can  be  performed  efficiently. 
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Consider  a  subcube,  //.  For  a  given  let  us  call  the  largest  hypercube  contained  in 

pvoc .  the  central  block  of  that  proc^Bi(H)^  and  denote  it  CB,-.  If  Bi(H)  has  two  largest 
hyperculjes  that  cannot  be  merged,  we  choose  the  lower-numbered  one. 

Lemma  3.2.1  Suppose  we  are  given  an  N -node  hypercube,  H,  holding  N  {g,v)-pairs  sorted  by  g- 
ralue.  ConsuUr  a  given  IJi(H).  If  2^'^-  —  1  >  Iproc )^  |  >  —  2  for  some  j,  llini  \CBi\ 

must  be  either  2^  or  2^'^^. 

Proof:  Cle.arly,  |C13i|  <  2^"^^ .  If  we  assume  that  |CBjl  is  2^*  for  h  in  the  range  0  <  /)  <  j  —  1, 
we  can  argue  to  a  contradiction.  Let  ns  divide  the  addresses  of  H  into  left  and  right  fields.  /  and  r. 
The  length  of  field  r  will  be  h.  The  address  of  CBj  is  of  the  form  LR,  where  L  and  R  are  values  for 
fields  I  and  i-,  respectively.  VVe  first  assume  L  is  even.  The  processor  at  location  (L  —  1)0*  can  not 
be  in  Bi,  because  then  CBj  would  have  an  address  of  the  form  (X  —  1)B.  The  processor  at  location 
(X  -I-  1)1*  can  not  l)e  in  Bi,  because  then  concatenating  the  two  hypercubes  (L)R  and  (X  +  1)B 
would  give  us  a  CBi  of  size  2*‘*'' .  Thus,  \proc^Bi{H)^  |  <  3  •  2*  —  2,  which  is  a  contradiction  for  all 
U's  in  the  range  specified. 

We  get  an  analogous  contradiction  if  X  is  odd.  | 

Lemma  3. 2.  J  is  used  in  Lemma  3.2.2  below. 


Lemma  3.2.2  Suppose  we  are  given  an  N-node  hypercube,  H ,  holding  N  (g,  v)-pairs  sorted  by 
g-value.  We  ran  reduce  every  Bi{H)  in  H  to  a  single  value  in  time  O(logN). 


Proof:  The  reduction  can  be  performed  in  three  steps: 

Step  1.  Each  processor  in  proc(^Bi(B)j  can  determine  if  it  is  in  CB,,  tis  follows.  In  parallel,  each 
processor  with  address  p  checks  the  y-values  of  the  two  processors  with  addresses  p  -I-  1  and 
p  —  1  to  determine  whether  it  is  either  the  first  or  last  processor  in  its  proc^Bi{H)j .  With 
two  distribution-from-leaders  operations^,  each  processor  can  receive  the  addres.ses  of  the  first 
and  last  proces.sors  in  its  proc(^Bi{H)j .  Using  the  results  of  Lemma  3.2.1,  we  know  that  if 

—  1  >  |proc^Bi(/f)j|  >  —  2  for  some  j,  then  |CBi|  must  be  either  2^  or  2^'*’^ 

Thus,  there  is  at  least  one  processor  in  proc(^Bi{H)^  with  an  address  having  j  or  more  trailing 
zeroes.  Let  us  say  that  such  a  processor  has  a  tP -suffix.  There  are  at  most  two  processors 
in  proc(^Bi(H)J  with  CP+'-suffi.xes.  (If  there  were  three  such  processors,  then  the  size  of 

pr(K(^Bi(H)j  wo\ild  have  to  be  at  least  2-'"*'^  -f  1.)  If  there  is  at  leeist  one  processor  with  a 
0'’''‘'-snffi.\,  then  the  lowest- numbered  such  processor  (call  it  p)  checks  to  see  if  the  processor 
with  address  p  -(-  2-'+^  —  1  is  a  member  of  CB;  .  If  processor  p  -|-  21'*’^  —  1  is  a  member  of 
CBj  ,  then  lCBi|  must  be  2^  +  '  and  p  is  the  first  processor  in  CB;  .  If  there  are  no  processors 
with  O-'  +  '-sulIixes,  then  |CB,|  must  be  2^  and  the  lowest-numbered  processor  witli  a  (P -suffix 


^The  dislribiilinn-froni-leaders  operation  was  introduced  in  [BH82].  A  number  of  the  processors  are  distinguislied 
as  leaders.  When  the  distribuliun-from-leaders  operation  is  performed,  the  leaders  share  tlieir  data  with  all  the 
processors  of  higher  arldress  up  to  b<it  not  including  the  nevt  'eader. 
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is  the  first  processor  in  CB,  .  This  information  can  be  transmitted  to  all  the  processors  in 
proc(^Bi(H)^  with  two  additional  distribution-from-leaders  operations. 

Step  2.  There  are  at  most  1CB,|  —  1  processors  in  proc(^Bi(H)^  before  the  first  processor  in  CB,.  If 
there  were  )CB,)  or  more  preceding  processors,  then  the  first  ICIbl  elements  would  constitute 
CB,-.  There  are  at  most  2-|CBj|—  1  elements  after  CB;;  otherwise,  CB;  would  be  twice  as  large. 
All  the  processors,  pj,  not  in  CB,,  send  their  triples  to  one  of  P;+|CB.|,  Pi-|CB.|.  o'"  P;-2  |CB,|, 
depending  on  which  of  these  three  is  an  address  in  CB,  .  The  processors  in  these  three  sets  can 
send  their  .triples  over  to  the  appropriate  processors  of  CBj  with  three  consecutive  monotone 
routing  steps. ^ 

Step  3.  Using  simple  tree  operations,  we  can  reduce  eacii  Bi{H)  to  a  single  pair,  in  parallel.  [B+Sfi] 
demonstrates  an  embedding  for  the  complete  binary  tree  with  ‘2"  —  1  nodes  in  a  hypercube 
of  size  2”.  This  embedding  has  dilation  and  load  factor  both  equal  to  2.  The  leaves  of  the 
embedded  tree  are  mapped  to  the  even-numbered  nodes  of  the  hypercube.  By  pairing  the  odd- 
and  even-numbered  nodes  of  the  hypercube  together,  we  can  perform  any  tree  operation  on 
the  hypercube  with  only  a  constant  factor  increase  in  running  time. 

In  Step  1,  each  processor,  p,  checks  the  ^-values  of  the  two  proce.ssors  p -f  1  and  p  —  1.  This 
checking  can  be  performed  simply  with  four  distribution-froni-leader.s  operations.  To  check  in  the 
forward  direction,  we  use  two  distribution-from-leaders  operations.  The  odd-numbered  processors 
are  the  leaders  in  the  first  such  operation  and  the.  even-numbered  processors  are  the  leaders  in  the 
second.  All  the  distribution-from-leaders  operations  used  in  thi.s  step  can  be  performed  in  time 
C>(logAt)  as  shown  in  [U1184].  The  monotone  routes  and  tree  operations  of  the  next  two  steps  also 
take  time  O(logN). 

I 

3.2.3  Hypercube  Implementation  -  Phase  1 

We  start  by  breaking  the  N  processors  into  N/S‘*  hypercubes  of  nodes.  If  S'*  >  N,  we  can 
perform  the  beta  operation  by  applying  Phase  1  to  the  entire  hypercube.  By  convention,  k  is  0  for 
the  one  step  of  this  phase;  thus  the  hypercube  addresses  are  of  the  form  uuVotuo^o  where  [uol  =  D—4, 
|t;o|  =  .3,  |wo|  =  0,  and  )a.-o|  =  1.  The  nodes  in  a  given  subcube  have  addresses  of  the  form  UqVq  A'q. 
For  each  subcube,  ff,  we  perform  the  following  three  stages: 

Sort  Stage.  Weu.se  odd-even  merge  .sort  to  sort  the  (<;,  t;)-pairs  in /f  by  j-value  in  lime  Tsort(-5‘*  ,  ■5'*). 
The  pairs  of  each  Bi(H)  will  now  reside  in  con.seculively  numbered  processors  in  H . 

Reduce  Stage.  We  now  invoke  Lemma  3.2.2  to  reduce  each  Bi(H)  to  a  single  triple.  This  invoca¬ 
tion  of  Lemma  3.2.2  takes  time  Oflog.S’). 

monotone  routing  step  is  a  permutatiun  of  the  items  such  that  if  tlieie  are  two  source  nodes,  cj  and  with 
two  corresponding  destination  nodes,  Sj  anri  ^2,  then  jtj  <  ,t2  implies  ,*>1  <  ,52-  int  iiitii'cly,  the  items  being  routed 
maintain  tlie  same  order  before  and  after  the  tnonotonic  route. 
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Compact  Stage.  At  the  conclusion  of  the  recluctioii  stage,  there  are  S  or  fewer  pairs  in  H .  Tlie 
pairs  represent  F  applied  to  the  iKvalues  in  each  separate  We  shall  call  these  the  final 

pairs.  For  each  final  pair,  we  conipule  how  many  other  final  pairs  are  in  processors  with  higher 
addresses  by  means  of  a  prefix  operation.  VVe  then  compact  the  final  pairs,  via  a  monotone 
route,  to  a  suhcnbe  of  size  S.  witli  address  of  the  form  Both  the  )>refix  operation  and 

the  monotone  route  can  be  performed  easily  in  time  O(logS). 

The  compaction  stage  can  be  performed  without  recourse  to  the  monotone  route  and  prefix 
operation.  Instead,  each  processor  that  does  not  contain  a  final  pair  can  set  its  jf-value  to  minus 
infinity.  Sorting  by  value  then  compacts  the  final  pairs  into  the  highest-numbered  proces.sors 
in  N,  as  above.  The  use  of  this  sort  does  not  increcise  the  asymptotic  time  complexity,  since  we 
have  already  sorted  once  during  the  sort  stage.  Nonetheless,  we  choose  to  ujse  the  monotone 
route  and  the  associated  prefi.x  operation  to  emphasize  the  fact  that  the  initial  sort  is  the 
only  operation  in  our  implementation  that  requires  time  Tsort (5^.  ■?'’)•  In  the  next  section, 
we  use  this  hyperciibe  implementation  as  a  paradigm  for  implementing  N,S)  on  other 

networks.  For  simplicity,  we  shall  use  sorting  in  the  compact  stages  of  these  impleniental  ions. 

3.2.4  Hypercube  Implementation  -  Phase  2 

Phase  2  is  performed  in  £)  -  4  steps.  At  the  beginning  of  Step  fc,  the  remaining  (g,t))-pairs  will  be 
broken  into  subcubes  of  size  5^  with  addres.ses  of  the  form  UkV^6'‘^.  Note  that  with  this  choice 
of  partitioning  the  hypercube,  each  subcube  contains  at  most  {g,  t))-pairs.  At  the  end  of  Step 
k,  the  remaining  (g,  v)-pairs  will  be  compacted  into  subcubes  of  size  S  with  addresses  of  the  form 
Uk^^'^^Xk-  Step  k  (1  <  t  <  £)  —  4)  is  performed  as  shown  below. 

Sort  Stage.  As  in  Phase  1,  we  consider  a  particular  subcube,  H,  of  size  5“*.  We  can  use  the 
Nassimi-Sahni  .sort  ([NS82])  to  sort  the  at  most  5^  {g,  u)-pairs  in  H  by  y-value.  Since  the 
number  of  processors  is  equal  to  the  .square  of  the  number  of  items  to  be  sorted,  the  sorting 
can  be  performed  in  time  0(iog5). 

Reduce  Stage.  In  this  stage,  we  shall  view  H  as  an  S~xS^  matrix  of  processors,  Uj  (see  Figure  3.3). 
The  column  number  shall  be  designated  by  the  two  least  significant  digits  in  vk  and  the  row 
number  shall  be  designated  by  the  remaining  digit  from  v*  followed  by  the  sole  digit  of  Xk- 
Initially,  a  single  row  contains  all  the  (j,  v)-pairs.  Using  a  prefix  operation,  we  can  determine 
which  processors  have  the  lowest  addresses  in  their  proc^Bi(H)j's.  We  shall  call  these  the 
leaders.  We  begin  by  broadcasting  the  contents  of  each  initial-row  proces.sor,  <i; ,  to  the  column 
j.  Next,  each  processor  tjj  l)roadcasts  to  row  j.  Finally,  in  the  columns  of  the  leaders,  F  is 
applied  to  those  i>-values  whose  corresponding  (/-values  match  the  leader’s.  The  four  tree 
operations  of  this  stage  can  ))e  performed  in  time  O(log.5). 

Compact  Stage.  We  consider  the  (.(/,t>)-pairs  that  remain  after  the  reduce  stage.  We  first  move 
all  of  these  pairs  to  the  initial  row.  Since  there  are  only  S'  groups  overall,  there  are  at  most  .S 
remaining  pairs  in  H  after  the  reduce  .stage.  In  time  0{log5),  we  can  route  these  pairs  into 
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tlie  highest  addresses  in  H  with  a  prefix  operation  and  a  monotone  route.  As  in  Phase  1,  we 
could  u.se  a  sort  to  effect  the  compaction. 

Phase  1  can  be  performed  in  time  0(Tsort(<S^ . +  logS).  Since  each  step  of  Phase  2  takes 
time  O(log5)  and  there  are  0(loglV/logS)  such  steps.  Phase  2  takes  time  O(loglV).  The  overall 
time  u.sod  l\v  (he  algorithm  is  thus 

0{\ogN  +  TsoKT{S^,S^)). 

Hy  Corollary  A. 3.1,  we  know  that  TsortC'^, 5”*)  is  15(7sort('S'.  •?))i  so  the  overall  time  can  be 
expressed  as  0(log  +  TsortC'^'.  5)). 

We  stated  that  if  >  N,  the  beta  operation  can  be  performed  by  applying  Phase  I  to  the  entire 
hypercube.  This  application  of  Phase  I  can  be  performed  in  time  0(loglV  +  TsokT‘{N,N)).  This 
bound  can  be  expressed  as  0{\ogN  +  Tsort(5,S)). 

Additionally,  using  the  above  implementation  as  a  paradigm,  we  can  perform  on 

the  butterfly  network  in  time  0(\ogN  +  Tsokt{S,S)),  and  on  the  shuffle-exchange  network  in  time 
0(\ogN  +  \og-  S). 

We  can  consider  also  a  variant  of  the  beta  operation  in  which  the  output  values,  yi,  are  not 
recjuired  to  be  sorted.  The  author  knows  of  no  algorithm  for  the  simpler  problem  of  competing  5, 
given  N  input  (^,  v)-pairs,  that  requires  time  less  than  C>(logM  Tsokt(S,S)).  Thus,  it  appears 
that  this  “un-sorted"  variant  of  the  beta  operation  is  no  easier  to  implement  than  the  ^\(N,  N,S) 
operation. 


3.3  Efficient  Mesh-of- Trees  Implementation 

In  this  section,  we  show  how  N,S)  can  be  implemented  on  a  mesh-of-trees  network  (MOT), 

using  the  hypercube  implementation  as  a  model. 

We  first  note  that  can  be  performed  easily  in  time  0{y/N)  on  a  y/N  X  y/N,  N- 

processor  mesh  system,  even  if  S  is  not  known  beforehand.  This  upper  bound  is  tight  since  there 
is  an  obvious  lower  bound  of  Q,{y/N)  time,  even  when  S  is  given.  In  the  case  of  MOT,  our  results 
are  obtained  for  the  \/N  x  y/N ,  0{N)  processor  MOT  system,  where  there  is  a  known  bound  on 
S.  As  was  the  case  with  the  hypercube,  we  shall  disregard  the  special  cases  when  divisions,  square 
roots,  and  logarithms  produce  non-integral  values.  Although  these  cases  present  no  special  problems, 
dealing  with  them  introduces  unnecessary  clutter.  We  shall  assume  that  the  processors  are  arrayed 
in  descending  row  major  order. 

3.3.1  The  General  Step 

d’he  general  step  in  Pha,ses  2  and  3  below  differs  from  the  general  step  of  the  hypercube  implementa¬ 
tion  in  two  ways.  Firstly,  the  size  of  tlic  sub-MOT’s  with  which  we  work  grows  in  each  step.  Recall 
that  in  each  step  of  the  hypercube  algorithm,  we  always  work  with  sub-hypercubes  of  a  single  size 
(S^  nodes  each).  Secondly,  each  sort-rediice-compact  sequence  is  preceded  bv  a  routing  stage. 
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Ill  Phase  1,  we  perform  /?i''‘(45, 45,  .S')  on  sub-MOT’s  with  side  \/45.  In  Phase  2,  we  incre2ise  tlie 
size  of  the  siib-MOT’.s  considered  until  the  number  of  processors  is  equal  to  the  square  of  the  number 
of  remaining  (flr,  t;)-pairs  in  each  sub-MOT.  Finally,  in  Phase  .3,  we  can  increeise  the  sub-MOT  size 
to  VN  X  ^/N  quickly. 

3.3.2  Auxiliary  Lemmas 

During  the  sort  and  compact  stages  of  Phase  2,  we  shall  sort  the  pairs  in  each  sub-MOT.  Before 
presenting  the  MOT  implementation  of  N,  .5),  we  shall  first  establish  several  auxiliary  lemmas 

and  theorems  that  demonstrate  that  this  sort  can  be  performed  efficiently. 

Lemma  3.3.1  An  arbitrary  partial  pemiulahon  routing  of  z  elements  that  starts  and  ends  on  the 
leaves  of  a  complete  binary  tree  with  m  leaves  can  be  performed  in  time  0(z  -\-\ogm). 

Proof:  Let  Sir  be  the  set  of  elements  that  needs  to  be  routed  from  the  left  half  of  the  tree  to 
the  right  half.  Sri,  Sii,  and  Srr  are  defined  analogously.  Using  pipelining,  we  can  easily  route  the 
elements  of  Sir  to  their  destinations  in  time  0(|5/,-|  -f  login).  Similarly,  the  elements  of  Sri  can 
bf  routed  in  time  0(|5r;|  +  logm).  To  route  the  elements  of  Sn,  we  actually  use  two  consecutive 
routings.  In  the  first  routing,  the  elements  are  routed  from  their  source  locations  in  the  left  half  of 
the  tree  to  the  first  )Sn)  locations  on  the  right  half.  The  root  will  keep  a  count  of  the  elements  as 
they  arrive  and  send  them  to  the  appropriate  destinations.  In  the  second  routing,  the  elements  are 
routed  to  their  final  destinations  on  the  left.  Both  routings  take  time  0{15ji|  +  login).  Similarly, 
the  elements  in  Srr  can  be  routed  in  time  Ofl.SVrl  +  login).  Since  2  =  |.Sr(l  -f  15;rl  +  15(il  -t- 15,.,.  1,  the 
overall  routing  can  be  accomplished  in  time  0(:  +  login).  | 

Lemma  3.3.2  Suppose  we  are  given  a  MOT  of  side  m,  with  all  elements  contained  in  the  first  z 
rows.  In  time  0(z  +  logm),  we  can  achieve  any  permutation  in  which  the  final  deslinattons  of  the 
elements  are  also  within  the  first  z  rows. 

Proof:  Let  R,  j  be  the  row  of  the  destination  of  the  element  that  starts  in  row  i,  column  j;  similarly, 
C'ij  is  the  column  of  the  destination.  We  apply  bemnia  3.3.1  three  times.  The  lemma  is  applied  fir.^t 
to  the  columns,  then  to  the  rows  and  finally  to  the  columns  of  the  MOT  so  that  each  element  from 
(i,j)  follows  the  permutations;  (i,j)  (i  +  j-j)  (f  +  j.O't.y)  — (Hij.Cij),  where  all  additions 
are  mod  m.  Note  that  the  first  application  of  Lemma  3.3.1  staggers  the  elements  so  that  no  row 
contains  more  than  z  elements  (see  Figure  3.4).  Each  of  these  three  permutation  operations  can  be 
performed  in  0(z  +  logm)  time,  yielding  the  desired  result.  | 

Theorem  3.3.1  Consider  a  MOT  with  side  in.  As.'nime  that  it  is  divided  vertically  into  two  halves. 
Assume  further  that  the  first  2  (1  <  r  <  m)  rows  on  the  left  side  contain  the  sorted  list,  A,  and  the 
first  z  rows  on  the  right  side  contain  the  sorltd  list,  B.  We  can  merge  these  two  lists  into  one  list 
located  in  the  first  z  rows,  in  time  T(z,m)  =  0(z  +  logm  log  2). 

Proof:  The  result  is  achieved  by  u.sing  odd-even  merge  (see  Figure  3.5). 
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(/  +  j,  Ci^j  )  -+  i^i,j  >  C't.j  ) 


Figure  3,4:  Permutations  of  Lemma  3.3.2. 
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Step  1.  We  start  l>y  separating  the  odd-position  elements  of  A  (.4„rf,y)  from  the  even-position  ele¬ 
ments  (.4  )  so  that  Acdd  occupies  the  first  c/2  rows  and  -4,,.,.,,  occupies  the  c/2  rows  starting 

at  row  m/2.  In  Figure  3.5,  we  shall  abbreviate  the  subscripts,  (e.g,,  Aodd  will  be  denoted  as 

Simultaneously,  we  separate  the  B’s. 

Step  2.  We  next  exchange  the  positions  of  Agdd  and 

Step  3.  We  now  want  to  merge  lists  that  are  stacked  vertically.  Consider  the  m/2  x  m/2  square 
in  the  upper  left.  We  separate  out  the  even-position  elements  of  Beven  {Beven-even)  and  the 
odd-position  elements  of  B^uen  (^ensn-odd)-  The  elements  of  B^ven-even  are  moved  to  the  left 
half  of  the  space  previously  occupied  by  B^ven  and  the  odd-position  elements  are  moved  to 
the  right.  At  the  same  time,  we  perform  the  analogous  separations  on  the  elements  of  Bodd, 
Af-ign  and  Aodd- 

Step  4.  We  next  exchange  Beven-even  and  Aeven-odd-  Simultaneously,  we  also  exchange  Bodd-even 
and  A  odd-odd- 

Stej)  5.  We  now  have  4  sub-MOT’s  with  side  m/2  and  c/2  rows.  We  can  recursively  perform 
merge  on  the  lists  in  these  sub-MOT’s  in  time  r(c/2,  m/2)  to  yield  the  four  lists  AB  even-even  > 
-A  B er-en-odd^  ABodd-even^  and  ABodd-odd- 

Step  G.  We  then  interleave  ABeven-even  with  ABeven-odd-  Using  only  swaps  of  adjacent  list  ele¬ 
ments,  we  can  produce  the  sorted  list  A  B even-  We  simultaneously  perform  the  same  interleave- 
swap  operation  on  ABodd-even  and  ABodd-odd-  yielding  the  .sorted  li.st  ABodd-  Lastly,  we 
interleave  ABeven  with  ABodd,  and  perform  any  necessary  value  swapping. 

Basis.  If  2  =  1  then  we  can  sort  in  time  O(logm). 

Induction  step.  Let  z  =  zq.  The  permutations  in  the  above  steps  can  all  be  performed  in  time 
c(c  4-  login)  for  some  constant  c  by  invoking  Lemma  3.3.2  and  using  simple  pipelining.  Thus, 
'/’(co.m)  <  r(2o/2,  m/2)  +  c'(co-|- login)  for  some  constant  c'  and  T{z,m)  =  ©(c-t-logmlogz). 

I 

Theorem  3.3.2  Consider  a  MOT  with  side  m.  Assume  ihai  there  are  0{mz)  elements  in  the 
first  z  I'ows.  He  can  sort  these  elements  with  the  results  ending  up  in  the  first  c  rows,  in  time 
7'(c,  III)  =  C>(c  -P  logmlog^  c). 

Proof:  We  use  a  merge  sort.  We  first  divide  the  MOT  into  four  sub-MOT’s  of  side  m/2. 
W'e  can  distribute  the  elements  into  the  first  c/2  row.s  of  each  .sub-MOT  with  simple  pipelining. 
We  then  recursively  sort  the  elements  in  each  sub-MOT  in  time  T(z/2,m/2).  We  next  merge 
the  four  sorted  lists  together,  using  the  methods  outlined  in  Theorem  3.3.1.  Hence,  T(z,m)  = 
T(z/2,  m/2)  -P  c{z  -P  logmlog  c)  for  some  constant  c.  Solving  this  recurrence  yields  the  time  bound 
claimed  above.  | 
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3.3.3  MOT  Implementation  -  Phase  1 

We  first  break  the  N  processors  into  N/4S  siib-MOT's  will)  side  \/45.  For  earli  snb-MOT,  .A/,  we 

perform  45,  S)  by  applying  the  following  three  stages. 

Sort  Stage.  We  first  sort  (he  pairs  in  each  sub-MOT  by  jy-value.  This  sort  can  be  accoiuplishcd  in 
time  0(^/S)  by  invoking  Theorem  3.3'2. 

Reduce  Stage.  Let  Bi(AJ)  refer  to  the  set  of  (y,i>)-pairs  in  M  from  a  particular  B, .  Using  tree 
operations,  we  can  combine  the  (it,  t))-pairs  from  each  Bi(M)  to  produce  one  repre.sentativc 
i')-pair  in  time  0(log5). 

Compact  Stage.  Each  processor  not  holding  one  of  the  resultant  pairs  .sets  its  ;/-valne  to  minns 
infinity.  A\'e  then  sort  again  on  (/-value,  so  that  the  remaining  (p,n)-pairs  are  compacted  in 
the  iirst  .b'  spaces  (in  the  row-major  sense). 


3.3.4  MOT  Implementation  -  Phase  2 

Phase  '2  is  [reiloriDed  in  (3s72)  —  1  steps.  y\t  the  beginning  of  Steji  k,  tlie  MO'f  is  divided  into 
,V/(4*'.S)  snb-MOT's  with  sides  of  length  \/4*5.  The  first  | >/.S74*'  j  rows  of  eacli  such  snb-MOT 
contain  il\e  ,'V  or  fewer  difTereni  (<j,  (')-pairs.  compacted  to  the  left.  For  convenienn'.  tln-se  initial  row.s 
of  the  snb-MO'l'  shall  henceforth  be  called  the  non-trivial-part  (NTP).  Step  k  (  \  <  k  <  3.s/2  -  1) 
IS  perlormed  witli  the  foilowing  sequence  of  stages. 


Route  Stage.  We  start  the  routing  stage  by  conceptually  clumping  4  conlignons  snb-.MOT's  into 
a  single  .s(|uarc  snb-MO  T  with  twice  the  side  length.  We  first  shift  up  tlie  N  TP's  of  iIk'  two 
lower  .suI.r-MO'T's  .so  that  they  are  contiguous  to  the  NTP's  of  the  upper  siiii-MO’T's.  'This 
i-csnlis  111  a  larger  snh-  .MOT  with  side  \/ 4*'+*5  having  a  NTP  occuiiying  Hw'  first  2  j 

rows.  'Thi>  loidiiig  can  he  done  with  simple  pipelining  in  time  0{  | \/5/4‘  "j ). 

Sort  .Stage.  We  then  sort  (his  nev\  .NTP  by  invoking  Theorem  .3  3.2,  The  sorting  ol  Step  k  can  be 
perrornied  in  time 

^/5/4*=]  -I-  log  \/4*  +  ‘.S'  log' 


O 


Reduce  Stage.  Tor  each  group  numi'er,  there  arc  up  to  four  ditferent  (r/,  ti)-pairs.  Wi  can  combine 
tliesi'  pair.--  into  mi"  (i;,i  )-pair  in  time  O(log  \/4*‘'  +  '.S'), 

Compact  Stag)-.  As  in  Phase  1,  we  compact  the  NTP  into  the  lirsl  5  spaces  (in  liie  row-major 
sensei  bv  iiii'.aiis  of  a  sort. 


Summing  tiie  lime  re.jnirements  of  each  step,  w.-  sec  that  Phase  2  can  Ix'  perlormed  in  lime 
/■I’/ 2  r  - - -n 


I  Y' 5,"P' I  -f  log  \/1' +'.‘)'log-  y5/4*' 
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=  0(\/S  +  log^S)  =  C»(v/5). 

A*  the  end  of  this  phase,  we  have  tV/i"*  sub-MOT’s  of  siile  5-,  each  with  no  more  than  S'  non-trivial 
(S,t>)-pairs. 

3.3.5  MOT  Implenifflitation  -  Phase  3 

Phase  3  is  performed  in  log{n/2s  —  1)  steps.  In  Step  L,  we  shall  increase  the  side  of  the  sub-MOT 
from  '■*■*  to  The  last  step  will  leave  n.s  a  .single  MOT  with  side  \/N.  Note  that  in  each 

sub-MOT,  the  number  of  processors  will  always  be  equal  to  the  square  of  the  number  of  non-trivial 
Step  Jb  (1  <  <  log(n/2.s—  1))  is  performed  as  shown  below. 

Route  Stage.  At  the  beginning  of  Step  k,  we  have  sub-MOT’s  of  side  S'*  each  with  no  more 
than  S  non-trivial  (g,  v)-  pairs.  VVe  shall  conceptually  clump  S'*  of  these  sub-MOT’s  into 
sub-MOT’s  of  side  S"*'*'^.  We  shall  view  each  such  sub-MOT  of  side  as  being  composed 

ofS^*'*  columns  of  width  5'*~*'*'^.  To  route,  we  move  tlie  NTP’s  from  the  sub-MOT’s  in  each 
column  into  the  diagonal  processors  of  that  column  as  a  preliminary  to  sorting.  We  follow  the 
lead  of  [U1184]  in  calling  the  proce.ssor  at  location  (».  /'),  the  itli  controller.  The  routing  can  be 
done  with  the  tree  connections  in  time  Oflogi'-*"^’) 

Sort  Stage.  For  each  clump,  we  have  a  sub-MOT  of.S-*^''*''  proce.s.sors  and  no  more  than 

non-trivial  (5,  t;)-pairs.  Since  the  number  of  proce.s.sors  is  at  least  equal  to  the  square  of  the 
number  of  pairs,  the  sorting  can  be  performed  in  time  0(logb'-*  +  ')  using  the  standard  MOT 
sorting  algorithm  [UI184]. 

Reduce  Stage.  The  reduce  stage  closely  resembles  the  standard  MOT  sorting  algorithm.  First, 
every  controller  checks  to  see  if  the  group  number  it  contains  is  the  leftmost  such  group  number. 
As  with  the  hypercube  algorithm,  we  shall  call  such  proce.ssors,  leaders.  Next,  each  controller 
bro^ulc^lsts  its  value  to  entire  row  and  column.  Finally,  in  the  columns  of  the  leaders,  F  is 
applied  to  the  u-values  whose  corresponding  {/-values  match  that  of  the  column’s  leader.  These 
tree  operations  can  be.  performed  in  time  (?(log5’'*'''' ). 

Compact  Stage.  Another  sort  will  then  compact  the  values.  As  in  the  compact  stage  of  Phase  2, 
it  is  assumed  that  non-leader  controVlers  have  (/-vahies  equal  to  minus  infinity.  The  sort  takes 
the  same  time  as  in  the  sort  stage  above. 

The  third  phase,  therefore,  takes  time  0(53'°=*”^*' *log S'*’*' )  =  0(u). 

The  overall  time  by  the  MO  T  algorithm  is  thus 

0{logN  -f  \/S). 

Note  that  for  both  the  mesh  and  the  .MOT,  =  Qi\/S).  Thus,  the  overall  time  required 

can  be  expressed  as  OflogA  +  7’sort(  '’'.  .S')). 
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3.4  Determining  the  Output  Size 

The  running  times  of  the  algoril  Inns  given  al>Gve  are  functions  of  lioth  the  input  size,  N,  and  the 
output  size,  5.  I'hese  algorithms  assume  that  .S'  i.s  known.  'I'lni.s,  tin'  question  arises,  what  do  we  do 
if  we  do  not  know  the  output  size,  S'? 

For  a  large  class  of  problems,  to  which  l)eia  operations  appear  to  belong,  the  problem  of  deter¬ 
mining  the  output  size,  5,  is  e.s.sent  inlly  as  complex  as  the  |>roblem  of  computing  the  output,  given 
the  output  size.  While  it  would  lie  possible  lo  develop  separate  algorithms  to  determine  the  output 
size,  we  shall  exhibit  below  a  gein'ral  scheme  that  enabh's  one  to  determine  .9  and  solve  the  problem 
in  time  proportional  to  that  required  for  solving  the  problem  alone,  given  S- 

3.4.1  Iterative  Guessing 

We  sha.!  create  an  algoritlim,  call  i(  Algorithm  B.  for  .solving  "P  given  output  siz.e 

S.'  We  shall  employ  a  strategy  of  ilcrative  gue.ssiug  that  consists  of  two  components.  Firstly,  'we 
require  an  algorithm,  call  it  Algorithm  A,,  for  'prol>lem  V  given  output  size,  S.’  Secondly,  we  reqi’.ite 
an  increasing  sequence  of  guess''^  foi  the  oiilpiil  size,  F  =  (</i .  ;/2, .  •  .)•  In  Algorithm  B,  we  shall 
sequentially  test  the  gues.ses  in  F  on  Algorithm  A.  That  is,  we  shall  first  run  Algorithm  A  with 
output  size  e<(ual  to  gi.  If  this  first  guess  is  too  small,  w’e  shall  run  Algorithm  A  with  outpu  ^  size 
equal  to //2,  then  ^3,  etc...  until  Algorithm  A  finally  succeeds. 

This  gue.ssing  strategy  will  bi-  ('(ficient.  if  Algorithm  A  and  sequence  F  possess  certain  properties. 
Algorithm  A  must,  satisfy  the  following  four  conditions; 

(i)  The  running  time  is  t{N,Q).  wlu're  Q  is  a  given  bound  on  the  output  .size. 

(ii)  If  <3  >  .9,  the  algorithm  works  correctly  and  produces  the  appropriate  output  of  size  S. 

(iii)  If  Q  <  S,  the  algorithm  can  detect  the  error  (within  timet(A,Q)). 

(iv)  f(N,Q)  is  monotonically  increasing  in  Q. 

bet  the  minimum  output  size  po.ssible  for  any  input  be  5,„in.  Sequence  F  must  satisfy  the 
following  two  conditions: 

(^)  go  —  ‘9niin 

(b)  3  Ci.ct  s.t.  I  <  C|  <  r-i  and  C; /( Ab  _r/,_ , )  < 

The  constants,  ci  ami  cn.  are  independent  of  i,  but  can  be  chosen  in  a  fashion  that  depends  on 
the  elements  of  F.  I'he  efRciency  of  the  iteralixe  guessing  strategy  is  demonstrated  in  the  following 
theorem. 

Theorem  3.4.1  Assume  lhal  at  an  (jucu  nu  algorilhm  for  ‘problem  V  given  S'  that  satisfies 
conditions  {i)-(iv).  Then,  if  we  arc  given  an  associate d  guess  sequence,  F,  satisfying  conditions  (a)- 
(/;).  we  can  create  an  algorithm  to  solve  "V  not  given  S',  in  time  t{N,S)  where  t(N,S)  =  Q{t(  N .  S)). 
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Proof:  The  given  algorithm  for  “P  given  5"’  is  our  ‘Algorithm  A  .  To  solve  the  problem  ‘P  not 
given  ,S”,  we  run  ‘Algorithm  R’.  Let  be  the  guess  for  Q  on  which  the  Algorithm  A  finally  succeeds, 
from  the  properties  of  Algorithm  A,  it  follows  that  gj-i  <  S  <  gj.  Hence, 


t{N,gf)  <  C2t(N,gf-i) 
<  C2t(N,S). 


Also, 


c\i(N,gi-i) 

J 

^cii(N,gi-i) 

1=2 

y-i 

tiN.gi)  +  ^(ci  -  l)t(N,gi) 

i=l 

/-I 

1  =  1 
/ 

'£tiN,9i) 

«  =  1 


S 

<  ^t(N,gi) 

t  =  2 


<  —^t(N,gj) 

Cl  -  1 

Cl  -  1 


From  the  definition  of  our  algorithm  for  ‘P  not  given  S\ 


iiN,S) 


< 

< 

< 


^t(N,gj) 

Cl  -  1 


Since  it  is  trivially  true  that 


tiN,S)  >  i{N,S) 

it  follows  that  iiN,S)  =  Q{t{N,S)).  Note  that  the  optimal  choice  of  C]  =  c-.  =  2  yields  a  factor  of 
4  slowdown  in  the  worst  case.  | 


3.4.2  Application  of  Method 

Lemma  3.4.1  Lcl  t(N,Q)  =  c(logAf  +  F{Q))  where  F{Q)  is  a  inonolouu  utly  increasing  funciiou 
and  c  ts  a  consinni.  The  guess  sequence 

Ua  =  1.  gi  =  min{x  |  F(x)  =  ((2'  -  1 )  log  .V)},  i  >  0 

satisfies  conditions  (a)-(6)  with  ci  =  cj  =  2. 
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The  algorithm  described  in  Section  3.2  satisfies  conditions  (i)-(iv)  and  the  associated  running 
time  is  of  the  form  describeil  in  Lemma  3.4.1.  Using  the  knowledge  that  S')  is  d  ■ 

TsortC-?, S')  for  some  constant  d  (as  shown  in  Appendix  A),  we  conclude  that  the  corollary  below 
follows  from  Lemma  3.4.1  and  Theorem  3.4.1. 

Corollai'y  3.4.1  pi'''[N,N,S)  can  be  tinplemcnted  on  a  hypercubc  tn  lime 

0(log  N  +  TsortI-S,  S')), 


withoid  prior  knowledge  of  S. 

The  algorithm  described  in  Section  3.3  also  satisfies  conditions  (/)-(it>)  and  the  a.s.sociatecI  run¬ 
ning  time  is  of  the  form  described  in  Lemma  3.4.1.  The  corollary  below,  therefore,  follows  from 
Lemma  3.4.1  and  Theorem  3.4.1. 

Corollary  3.4.2  N,S)  can  be  implemented  on  a  MOT  in  time 

0{\ogN  +  \/S). 

without  prior  knowledge  of  S. 

Recall  that  since  Tsort{S,  S)  =  0(\/S)  for  the  MOT.  the  time  for  (he  MOT  implementation  of 
N,  S)  can  be  expressed  as  0(log N  -t-  Tsort(S'. .S')). 


3.5  Lower  Bounds 

In  tliis  section,  we  shall  prove  some  lower  bounds,  given  our  formulation  of  the  beta  operation,  and 
relate  them  to  the  areas  and  times  associated  with  the  algorithms  and  architectures  discu.ssed  above. 
Note  that  while  in  the  other  sections  of  this  chapter  we  use  the  word  model  of  computation,  In're 
we  use  the  bit  model  of  computation. 

The  input  consists  of  N  pairs  of  numbers,  (30. 'n)- ('/i ,  I'l ),...,  (r/A  _ i ,  i',v- 1 ).  VVe  shall  demon¬ 
strate  the  lower  bound  for  the  case  where  these  numbers  are  constrained  to  be  in  the  range  0  to 
N  —  1.  Recall  from  Section  '2.1  that  =  {t»j  |  t.y  6  Zl,}.  For  all  i  such  that.  |/,1  >  0,  we  outinit  (/,  i/i) 
sorted  by  j,  where  y;  is  the  F-reduction  of  Recall  also  that  G  =  {/|  |il,|  >  0}.  Let  wd  be  the 
smallest  member  of  G;  similarly,  let  w,  (i  <  jG'l)  be  the  (/  +  l)-(h  smallest  member  of  G  Define 
M  =  Vw,  ',  that  is,  Zi  is  the  y-value  of  the  (?-|-  l)-th  output.  We  shall  refer  to  the  j-th  bit  of  Zi  as  Zij. 
(Similarly  for  the  g's.) 

3.5.1  A  Lower  Bound  on  Area 

The  structure  of  the  following  lower  bound  proof  follows  closely  that  of  the  proof  that  appears  in 
Examjrle  2.3  in  [U1184].  We  begin  by  introducing  the  following  lemma. 


5.5.  LOWER  BOUNDS 
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Leiuma  3.5.1  In  a  when-  and  where-delenntnaie  circuii  that  performs  the  beta  opemtion  correctly 
for  any  S,  none  of  the  output  bits  Zij  (fori  <  N  —  \)  can  be  output  before  all  of  the  input  bits  fiij 
(for  j  >  0^  have  been  read. 

Proof:  Assume,  to  the  contrary,  that  Zp  ,,  (p  <  N  —  \)  'is  output  before  ^ui,r  >  0)  ..s  read.  VVe 
construct  two  possible  inputs  by  fixing  every  g  and  v,  except  gm,  as  follows: 

•  Choose  nt  ^  w.  Set  =  p  +  1;  t’t  =  2*. 

•  Set  all  other  Vi  =  0. 

•  For  all «  (other  than  i  =  t  and  i  =  tc),  choose  a  value  of  gt  (different  from  p  and  p  +  1)  in  such 
a  way  that  Vj,  0  <  j  <  p,  3r  such  that  =  j. 

The  two  possible  inputs  yielded  by  setting  either  po,  =  p  or  pu,  =  p®  2*  produce  different  values 
for  the  bit  Zp^g.  Yet  Zp^g  is  output  before  is  read  -  contradiction.  | 

Theorem  3.5.1  Any  when-  and  where-determinate  circuit  that  can  perform  the  beta  operation  cor¬ 
rectly  for  any  S  must  have  A  =  Q(N  log  .Y). 

Proof:  (This  proof  assumes  N  is  even.  The  proof  for  N  odd  is  similar.)  We  shall  construct  a 
family  of  inputs  of  size  (Y/2)!,  each  with  different  outputs.  For  all  i,  fix  =  i.  For  i  >  N/2,  fix 
=  2t  —  Y  +  1.  Now,  we  allow  the  remaining  inputs,  go,  ■  ■  ■,gN/2-i  to  be  any  permutation  of  the 
even  numbers  less  than  N . 

Focus  on  the  time  just  before  the  first  r,j  (for  »  <  N  —  1)  is  output.  The  circuit  has  already 
read  all  of  the  bits  that  differ  between  the  elements  of  our  feunily  of  inputs.  Hence,  all  inputs  read 
subsequently  will  be  the  same  for  any  element  of  our  family.  Since  the  circuit  must  still  produce 
{N/2)\  different  outputs,  it  must  have  at  least  {N/2)l  states,  and  hence,  at  least  log((N/2)!)  = 
n(YIogW)  bits  and  area.  | 

3.5.2  An  Lower  Bound 

The  structure  of  this  lower  bound  proof  closely  follows  that  of  the  lower  bound  proof  in  Section  5 
of  [Hoc85]. 

Theorem  3.5.2  Any  when-  and  where-determinate  circuit  that  can  perform  beta  operations  for  any 
S  requires  AT^  =  Q(NStog  N),  where  N  is  the  number  of  input  pairs. 

Proof:  If  there  is  an  input  pad  that  reads  bits  of  the  u,’s,  then  T  =  Q{S^^').  Tliis 

condition,  coupled  with  Theorem  -3.5.1,  implies  our  AT^  bound. 

If  no  pad  read.s  bits,  then  we  may  partition  our  circuit  into  a  set  of  blocks  B  vvith 

|B)  =  so  that 

•  each  block  in  B  has  area  )  and  perimeter 
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•  each  block  in  B  reads  al  niosl.  0{S)  bits  of  the  in's. 

Sucli  a  collection  of  blocks  is  obtained  by  first  cutting  the  circnii.  into  Q(  )  l)locks  each  of 

l)eiiniet('r  (Lemma  2.3  of  [U1I84]).  Then  another  0(  -  )  cuts  suffice  to  ensure  that 

each  resulting  block  reads  at  most  /max  =  0(S)  bits  of  the  in's. 

Two  statements  are  true  of  the  above  set  of  blocks: 

•  At  least  half  of  these  blocks  produce  less  than  twice  the  average  number,  0(^),  of  the  output 
Zj  bits  for  i  <  S. 

•  laM.  /avr  be  defined  as  N  log  A715|  =  Q(5).  More  than  half  of  the  blocks  read  at  Iccist  2/av«.  — 
/max  input  bits. 

Note  that  we  can  stay  within  our  block  cutting  rules  and  still  make  our  blocks  small  enough  so  that 
2/avf~  /max  =  fi(S).  Thus,  there  is  some  block,  b  £  B,  that: 

•  reads  at  least  /i  =  f2(.S')  input  bits  from  of  the  i;,  ’s  (assume  without  loss  of  generality  that 
these  are  from  cq,  . . . , 

•  ha.s  perimeter  0{yj 

•  produces  1.3  =  O(^)  of  the  Zi  output  bits  with  i  <  5. 

VV(?  may  then  construct  a  fooling  set  as  follows.  Let 

=  {(*,  j)  I  h  reads  in  bit  j  of  r,  } 

Z  —  {i\b  outputs  a  bit  of  Zi,  and  i  <  5). 

For  each  1  <  '  <  /•_>,  choo.se  a  distinct  o,-.  such  that  Oi  ^  Z  and  0  <  Wj  <  S.  (Note  that  we  can  stay 
within  our  block-cutting  rules  and  still  make  our  blocks  small  enough  so  that  l->  +  h  <  S  —  1.)  Let 
c  —  S  —  12-  Choose  di, . . .  ,//c  (each  /?  <  S)  to  be  distinct  from  each  other  and  the  Ui's.  Choose  the 
//,  ’s  as  follows 


=  di. 

for  i  =  1  to  I2 

.7i+h 

=  A, 

for  r  =  1  to  c 

-  A. 

for  all  other  ;. 

And  the  c,  s  as 

bit  j  of  Hi  =  0  or  1,  for  (i,  j)  € 
bit  j  of  Vi  =  0,  for  (i,j)  ^  V 

Each  of  our  2‘'  choices  for  the  input  yields  a  different  configuration  of  the  output  bits  outside  of  b. 
As  l\  =  the  fooling  set  is  of  size  This  yields  the  bound  y^;vi^g  at ^  “  ^{S)\  that  is, 

A7’- =  «(.V,S'logA').  I 

In  Section  3.3,  we  showed  that  the  problem  can  be  solved  in  time  0('/S  +  \ogN)  on  a  MOT.  The 
resulting  AT-  bound  of  Nlog^N(.9+  log- JV)  is  within  a  few  log/V  factors  of  the  lower  bound 
of  A7'-  =  0(  N.9  log  N)  shown  above,  even  after  accounting  for  the  word  model  vs.  bit  model 
distinction. 


Chapter  4 

Generalized  Beta  Operations 


4.1  Overview 

In  CHiapter  3.  we  considered  the  efficient  implementation  of  P,  S).  In  this  chapter,  we  present 

efficient  implementations  of  P,S)  for  all  possible  relationships  among  S,N,  and  P.  We  shall 

concentrate  on  the  hypercube  network. 

We  first  note  that  the  number  of  groups  represented  in  the  pairs  can  be  no  larger  than  the  number 
of  pairs  (i.e.,  5  <  N).  Thus,  there  are  three  possible  relationships  among  the  variables  S,N,  and  P. 
We  shall  denote  these  relationships  as; 

Case  1;  S  <  N  <  P 

Case  2:  S  <  P  <  N 

Case  3:  P<S<N 

In  Sections  4.2,  4.3,  and  4.4,  we  shall  deal  with  the  question  of  how  to  implement  P,  S) 

efficiently  in  these  three  cases,  when  S  is  known.  The  running  times  of  these  implementations  are 
all  expressed  in  terms  of  TsoKr{f^,P)-  In  Section  4.5,  we  provide  actual  time  bounds  for  the  three 
cases.  In  Section  4.6,  we  shall  generalize  the  result  of  Section  3.4,  demonstrating  that  the  same  time 
bounds  can  be  achieved  without  knowing  S  beforehand. 

We  shall  follow  the  convention  that  the  S  output  pairs  end  up  sorted  by  (/-value,  [5/P]  per 
processor,  in  the  first  E  processors,  where  E  =  Minimumf.S,  P).  In  what  follows,  we  shall  let  N/P 
be  called  R.  We  shall  also  call  log  R,  r. 

4.2  Case  1:  S  <  N  <  F 

Theorem  4.2.1  Let  Z  —  Miniiniim(S^ ,  N).  If  tre  ate  given  M  [g ,  c)-patrs,  (listributcd  one  or  fewer 
per  processor,  in  a  P-processor  kyperenbe,  then  P,  S)  can  be  performed  in  Ume 

0(\ogP  +  TsoKr{Z,^)j 
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taken  P  >  N . 

Proof:  This  result  is  obtained  easily  by  modifying  the  Section  3.2  implementation  of  P,  S). 

We  consider  two  possible  cases. 

[  Z  =  ]  Recall  that  aside  from  the  .sort  stage  of  Phase  1,  the  Section  3.2  implementation  of 

P,  S)  requires  time  0(log  P).  Since  Z  =  (i.e.,  <  N),  it  suffices  to  show  that  the 

sort  stage  of  Phase  1  of  the  Section  3.2  algorithm  can  be  performed  in  time 

0(logP  +  TsoRT(5^^)). 

We  start  by  re<.listributing  the  pairs.  With  the  use  of  a  simple  prefix  operation,  each  pair 
finds  the  number  of  pairs  with  higher-numbered  addresses.  Using  a  monotone  route,  we  can 
route  the  pairs  so  that  each  set  of  pairs  is  contained  in  a  suboibe  with  S^/R  processors. 
The  pairs  in  each  subcube  can  then  be  sorted  in  parallel  in  time  Tsort(S^ ,  / R) .  Using  one 
more  prefix  operation  and  monotone  roule,  we  can  compact  the  pair.s  in  the  N  highest-ad dress 
processors.  The  monotone  routes  and  prefix  operations  can  be  performed  in  time  O(IogP). 

(  Z  =  W  ]  We  start  by  sorting  all  the  pairs  in  time  Tsort(^.  P<S)  then  be  com¬ 

pleted  using  the  reduce  and  compact  stages  of  Phase  1  of  the  Section  3.2  algorithm  ap¬ 
plied  to  the  entire  liypercube.  Since  Z  —  N  (i.e.,  S’®  >  N),  the  total  time  required  is 
0{\ogP  +  Tsorv{Z,  Z/R)). 

I 

4.3  Case  2:  S  <  P  <  N 

In  this  section,  we  deal  with  the  question  of  how  to  implement  0i'^{N,  P,S)  efficiently  in  the  ca,se 
where  S  <  P  <  N .  We  begin  by  introducing  two  simple  auxiliary  lemmas: 

Lemma  4.3.1  We  can  perform  {N ,  I,  S)  in  lime  N\ogS. 

Proof:  We  shall  keep  a  balanced  search  tree  of  (g,  u)-pairs  representing  (/-values  encountered  .so  far. 
We  visit  the  N  pairs  in  turn.  For  each  unvisited  pair,  (gi,Vi),  we  first  check  whether  group  ;/,  is 
represented  in  the  tree;  if  so,  we  combine  the  value  v,  with  the  associated  I’-value.  If  r/,  is  j)ot  in 
the  tree,  then  we  insert  tlu'  pair  (fii,Vi)  into  the  tree.  The  time  required  to  update  the  trc'e  in  either 
case  is  0{\ogS).  The  time  to  process  all  N  pairs  is  thus  O(JVlogS).  | 

Lemma  4.3.2  Suppose  we  are  given  a  P-nodc  hgpercube,  with  each  processor  holding  two  (g.  v)- 
pairs.  We  can  perform  j3\'^(2P,  P,S)  in  time 

O(log  P  -I-  Tsort(5,S)) 


taken  S  <  P. 


4.:l  CASE  2:  S  <  P<  N 
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PJroof:  We  start  by  performing  P,S)  once  on  the  first  pairs  of  each  processor  and  then 

again  on  the  second  pairs.  Let  us  call  the  pairs  remaining  from  the  first  operation,  A.  Let 

us  call  the  pairs  held  by  the  |-4|/2  highest-numbered  proce.ssors,  Ah  (respectively  for  the  other 
pairs  of  A).  The  output  pairs  resulting  from  the  second  0i'^{S)  operation  will  be  termed  Bh  and 
Bi,  analogously. 

Consider  the  following  series  of  operations  on  and  Bh-  Ah  and  Bn  are  lirst  merged  by 
(/-value.  Each  processor,  p,  holding  a  pair  from  Ah  then  checks  to  see  if  either  neighboring  proces.sor 
is  holding  a  pair  from  Bh  with  the  same  flr-value.  If  there  is  such  a  neighboring  proces.sor,  q,  the 
1/- values  of  the  two  pairs  are  combined  and  placed  in  the  n-slot  of  p’s  pair.  The  ((/,  e)-pair  of  q  is 
then  deleted.  Finally,  the  initial  merge  is  reversed  on  the  pairs  of  Ah  and  the  remaining  pairs  of 
Bh-  As  a  result  of  these  operations,  no  (/-value  appears  in  a  pair  of  both  Ah  and  Bh  Note  that 
this  series  of  operations  can  be  performed  easily  in  time  0(log  P). 

The  above  operations  are  then  repeated  on  A//  and  Bi,  Ai  and  Bi,  and  Al  and  Bh-  Following 
these  operations.  Ah,  Al,  Bh,  and  Bl  are  disjoint  with  respect  to  their  (/-values.  We  can  then  com¬ 
plete  0i^{2P,P,S)  by  performing  l3x~{P,P,S)  on  B  followed  by  0i^{P,  P,S)  on  all  tlie  remaining 
pairs.  Since  0i(S)  is  performed  a  constant  number  of  times,  the  overall  time  required  is 

0(log  P  -I-  Tsort('?,  S})- 

I 

There  are  two  possible  subcases:  either  R>  S  or  R  <  S.  We  shall  consider  these  two  possibilities 
in  turn. 

4.3.1  R>S  {N>  PS) 

Theoi-eni  4.3.1  If  we  are  given  N  pairs  distributed  evenly  ai  ike  P  processors  of  a  hypercube  and 
R>  S,  then  we  can  perform  0i'^(N,P,S)  in  time 

0(/?logS  -t-  logF  -t-  7sort(5',  5)) 

when  S  <  P  <  N . 

Proof: 

[Phase  1]  We  start  by  invoking  Lemma 4.3.1  to  perform  0-[^(R,  I,.?)  in  parallel  at  each  processor. 
/?!  '*'(/?,  1 ,  S)  takes  time  C)(/i  log  5).  There  are  now  at  most  S  remaining  pairs  at  each  processor. 
Note  that  these  remaining  (j',  t;)-pairs  are  now  in  sorted  order. 

[Phase  2]  We  now  group  the  processors  into  subcubes  of  size  .S'.  Consider  a  particular  such  subcube, 
H .  We  repeat  the  following  sequence  of  three  steps  on  all  the  sulicubes  in  parallel  until  Lliere 
are  no  more  (3,  a)-pairs  to  be  considered: 

1.  Find  the  minimum  (/-value  among  the  remaining  ((/,  e)-pairs  in  H . 
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2.  Find  the  result  of  applying  F  to  the  list  of  r- values  of  all  the  (y,  t>)-pairs  wi(  h  this  3- value 
and  renifive  these  pairs. 

3.  Send  the  resulting  pair  to  the  highest-numbered  j)roce.s,sor  in  H  not  yet  holding  a  repre¬ 
sentative  pair. 

Since  the  initial  '*'(/?.,  1,5)  leaves  the  pairs  sorted  at  each  processor,  each  step  in  Phase  2  can 
be  accompli.shed  with  a  simple  tree  operation.  Thus,  the  entire  phase  takes  time  0(51og5). 
After  this  phase,  there  is  at  most  one  (3,u)-pair  at  each  processor. 

[Phase  3]  W^e  then  perform  the  Section  3.2  algorithm  for  computing  P,S).  This  algorithm 

requires  time  O(log  P  +  Tsort(5, 5)). 

By  adding  the  time  requirements  for  each  phiise,  we  can  see  that  the  overall  time  required  is 

0(FlogS -t- log  F -f  Tsort(5,  5)).  | 

4.3.2  R<S  (N  <  PS) 

Theorem  4.3.2  Lei  Z  =  Minim7im{S- ,  N ).  If  we  arc  given  N  pairs  distributed  evenly  at  the  P 

processors  of  a  hypercube  and  R.  <  S,  then  we  can  perform  fii'^{N,P,S)  in  time 

O  ( R  log  5  +  log  P  4-  TsoKriZ.  Z/R.)) 

when  S  <  P  <  N . 

Proof: 

VVe  consider  two  possible  cases. 

[  Z  =  5'  ]  We  start  by  grouping  the  processors  into  subcubes  of  size  S^/R.  Consider  a  particular 
such  subcube,  H .  We  shall  perform  the  following  three  phases  on  each  subcube  in  parallel; 

[Phase  1]  Since  there  are  R  pairs  at  each  proces.sor,  there  are  S~  pairs  in  H.  The  pairs  of  H 
can  be  sorted  in  time  7sort(5-,  5'//?). 

[Phase  2]  Each  processor  in  H  performs  5i ’’’( P,  1 , 5)  on  the  pairs  it  contains.  Since  the  pairs 
are  already  sorted,  this  operation  can  be  performed  in  time  linear  in  the  number  of  pairs. 
Reasoning  eis  below,  we  can  conclude  that  after  this  beta  operation  is  performed,  there 
are  no  more  than  25“/P  pairs  remaining  in  H . 

(g,  l^e  one  fewer  than  the  number  of  contiguous  processors  in  H  that  contain  pairs 
from  Pi,  after  the  sort  of  Phase  1.  After  the  initial  1,  S),  the  number  of  remaining 

pairs  representing  group  iji  is  -|-  1. 

'E^c,<syR  =  ]H\ 

s.ec; 


Y.(^a.  +  ^)<syR+s  <2SyR 

g.^G 


Hence, 


■1.4.  CASE  3:  P<S<N 
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[Phase  3]  After  Phase  2,  there  are  are  no  more  than  2S^ /R  pairs  left  in  H ,  and  no  more  than 
R  pairs  left  in  any  given  processor.  Ulhnan  ([Ull])  showed  that  in  time  0{R\ogS),  the 
pairs  of  If  can  be  lecli.stribnted  so  that  there  are  no  more  than  two  pairs  per  processor. 

To  complete  the  implementation,  we  now  perform  Pi'^('2P,  P,S)  on  the  whole  hypercube  by 
invoking  Lemma  4.3.2.  "J  lie  total  required  time,  expressed  in  terms  of  Z,  simplifies  to 

0  (/?  log  S  +  log  P  +  Tsort(2)  Z/ P))  . 

[  Z  N  \  If  Z  =  N  (i.e.,  S-  >  N).  P,S)  can  be  performed  by  applying  the  above  three 

phases  to  the  whole  hypercube.  Phase  1  requires  time  Tsort(7V',  P).  Phase  2  requires  time 
0(P),  as  before.  After  Phase  2,  there  are  at  most  P  +  5  <  2P  remaining  pairs  and  no 
more  than  R  pairs  left  in  any  given  processor.  The  redistribution  of  Phase  3  requires  time 
O(PlogP).  Since  >  N  >  P,  logP  =  O(logS),  so  Phase  3  requires  time  0(Plog5).  The 
final  /9i'*'(2P,  P,  5)  requires  time  0(logP  +  Tsort(5,  S)),  as  before.  The  total  required  time, 
expressed  in  terms  of  Z,  simplifies  to 

0{R\ogS +  Tsokt{Z,Z/R))  . 
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4.4  Case  3:  P  <  S  <  N 

In  this  section,  we  deal  with  the  question  of  how  to  implement  {N ,  P,  S)  efficiently  in  the  case 
where  P  <  S  <  N .  As  in  Section  4.3,  there  are  two  possibilities:  R  >  S  and  R  <  S.  We  shall 
con.sider  them  in  turn. 

4.4.1  R  >  S  (N  >  PS) 

Theorem  4.4.1  If  we  are  gtven  N  pairs  distributed  evenly  at  the  P  processors  of  a  hypercuhe  and 
R  >  S.  then  we  can  perform  {N ,  P,S)  in  time  0(Plog5)  when  P  <  S  <  N. 

Proof:  The  algorithm  for  this  subcase  mirrors  the  algorithm  of  Section  4.3.1. 

[Phase  1]  We  start  by  invoking  Lemma  4.3.1  to  perform  Pi'^{R,  1,S)  separately  at  each  processor. 
This  operation  takes  time  0{R]ogS)  and  reduces  the  maximum  number  of  pairs  at  each 
procc-s.sor  to  S.  Note  that  the  remaining  (.9,  t>)-pairs  are  now  in  sorted  order. 

[Phase  2]  W  e  now  perform  the  following  series  of  steps  repeatedly  until  there  are  no  more  (</,  1)- 
pairs  to  be  considered; 


1.  Find  the  minimum  i;- value  among  the  remaining  {g,  i))-pairs  in  the  whole  hypercuhe. 
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2.  FinJ  the  result  of  applying  F  to  the  list  of  w-values  of  all  the  (1/  ii)-paii«  with  this  (/-value 
and  rf'iiiovc  those  pairs. 

3.  Send  the  resulting  pair  to  the  highest-numbered  processor  not  yet  holding  \S/ P'l  lepre- 
■sentative  pairs. 

Since  the  initial  (R,l,S)  leaves  the  pairs  sorted  at  each  processor,  each  step  in  Phase  2  can 
be  accomplished  with  a  simple  tree  operation.  Thus,  the  entire  phase  takes  time  <9(5  log P). 

Since  S  <  R.  and  P  <  S,  SlogP  =  (9(Plog5).  Thus,  this  subcase  can  be  performed  in  time 
0(7?  log  5). 

I 

4.4.2  R<S  {N<PS) 

Theorem  4.4.2  If  we  are  given  N  pairs  distributed  evenly  at  the  P  processors  of  a  hypercube  and 
R<  S,  then  we  can  perform  l3\'^{N,P,S)  in  time 

0{R\ogP  -b  TsoKriN,  P)) 

■when  P  <  S  ■<  N . 

Proof;  The  algorithm  for  this  subcase  is  patterned  after  the  Z  =  N  subcase  of  Theorem  4.3.2. 
[Phase  1]  VVe  begin  by  i>eiTorming  SORT(A^,  P). 

[Phase  2]  Each  processor  now  performs  1,5)  in  parallel.  Since  the  pairs  are  already  sorted, 

each  proces.sor  requires  time  only  linear  in  the  number  of  pairs. 

[Phase  3]  Following  the  /?i^(P,  1,5)  operation  of  Phase  2,  there  are  no  more  than  P  -f  .S'  <  25 
remaining  pairs  and  no  more  than  R  pairs  in  any  given  processor.  We  can  reduce  th'*  total 
number  of  pairs  to  5  with  no  more  than  R  in  any  given  proces.sor.  as  follow.-.:. 

Let  us  call  the  smallest  //-value  contained  in  a  pair  in  the  processor  with  address  ay,,  and 
the  largest,  a.;,.  Call  the  associated  pairs  a-pairy,  and  (a-pairy,,  respectiv('ly.  After  the  first  two 
pha.ses  are  completed,  a-pairy,  and  ra-pairy,  are  the  only  pairs  in  py,  that  can  belong  to  groups 
that  appear  in  pairs  outside  of  pyy.  Hence,  we  shall  cal!  these  two  pairs  rcducihU.'  Wo  shall 
call  the  pairs  that  are  not  reducible,  irreducible. 

Let  us  view  the  reducible  pairs  as  a  string  of  length  no  greater  than  2P.  This  string  is 
composed  of  contiguous  substrings  of  pairs  with  common  g-values,  Wi'  can  compact  each 
substring  to  a  single  pair  in  parallel  in  time  O(logP)  using  the  method  of  Lemma  3  2.2  After 
this  compaction,  there  is  exactly  one  representative  pair  from  each  group  in  the  liypercube. 

*  ft  may  that  there  is  only  one  reducible  pair  at  a  processor. 
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[Phase  4]  There  are  now  5  leniainiiig  pairs  and  no  more  than  R  ]>airs  left  in  any  given  processor. 
Uliman  ([fill])',  showed  that  in  time  0(2?  log  P).  thest'  pairs  can  be  distributed  f5/P]  per 
processor. 

[Phase  5]  We  complete  the  operation  by  sorting  the  pairs  by  (/-value. 

The  total  required  time  for  the  five  phases  simplifies  to 

0(22 log  P  +  Tsort(22,  P)). 


I 


4.5  Time  Analysis 
4.5.1  Case  1:  S  <  N  <  P 


Theox-ein  4.5.1  If  we  are  given  N  pairs  dislribuUd  eve  nig  al  the  P  processors  of  a  hgperciibt,  then 
we  can  perform  P,S)  in  lime 


O 


J2£jL^ 

log  P/ AT/ 


when  S  <  N  <  P. 


Proof: 

In  [NS82],  it  is  shown  that  if  we  have  a  P-processor  hypercube  and  N  items,  where  P  =  22*+'/2 
for  j  in  the  range  1  <  y  <  logfV,  then  we  can  sort  the  items  in  time  0(y  log2V). 

We  know  from  the  proof  of  Theorem  4.2.1  that  P,  5)  can  be  performed  in  time 

O  ^logP  +  Tsort('^. 

where  Z  =  Minimum(5^,  A/). 

If  53  <  AT,  then 

Tsort(^.  -j^)  -  '2sort(5'^>  ^). 

Let  P'  =  S^/R  and  N'  =  5^.  P'  is  tlien  (’qual  to  (A'')'  +  ’2T  where  f  —  log5'^/(log P/A').  Dy  the 
[NS82]  result,  SORT(S^,S^/P)  (=  SORT(;V',  R'))  can  be  performed  in  time 


0(,/'log,s')  =  O 


10^-6'  \ 
log  P/Nj  ■ 


If  S3  >  (V,  then 


Tsoin  '-)  -  Tsort(A^,  P). 
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By  the  [NS82]  result,  7sor'i  (A'.  /’)  is 


This  bound  can  also  he  expressed  as 


Thus,  the  overall  lime  required  is 


O 


I 


4.5.2  Case  2:  5  <  P  <  N 

Theorem  4.5.2  If  irc  arc  yirf  n  N  pairs  distrihuled  evenly  a1  the  P  processors  of  a  hypercube,  then 
u>e  can  perform  P,,S)  in  time 

0(  R  log  .s'  +  log  P  +  —  log"  5) 
r 

wh en  S  <  P  <  .V . 

Proof:  In  Section  -1 .3. 1 .  it  was  shown  lliat  if  R.  >  .S,  we  can  perform  P,  S)  in  time 

0(  /f  logS  +  log  P  +  7'sort(-'5.  S)) 

when  5  <  /'  <  A  Since  fsou  i  (  s.  A )  =  O(log'.S')  and  log5  <  S  <  R,  the  above  expression  can  be 
simplified  to 

0(/nog.S  +  logP). 

In  Section  d..3.2,  it  wa.s  shown  that  if  R  <  S  we  can  perform  /3i'^{N,  P,  S)  in  time 

O  (  R  logs'  +  log  P  +  TsortC^.  Z/R)) 
where  Z  =  Minimum! .S'- ,  A')  and  .S'  <  P  <  ,.V. 

[(  SSS]  demonst raU'  an  algoni  hm  (or  sorting  A  items  on  P  processors  when  N  =  P^  +  O''-  for  a 
constant  c.  [C',S'88|  also  demonstrate  how  tlieir  algoritlirii  can  Ite  implemented  on  a  sliuffle-e.vchange 
network  in  time  0{r  /’'A  log/’)  and  on  a  hypermbe  in  time  0(c' P'/' log  P).-  In  Appendix  A, 
we  show  that  tin'  hypercnln'  sort  can  also  be  implemented  in  lime  0{c  P'/'^logP),  using  a  simple 
modiRcation  of  the  [CSSS]  results  '  These  .sorts  require  that  N  =  P'  +  '/c  for  a  constant  c  >  1/2, 
and  that  P  >  (4r-  -1-  c)-^. 

^  Allhougli  .  is  ,1  I  Iiiisiniil .  sli.ill  111,1  <■  it  i.-.Nplii  iilv  in  tin-  big-O  nnlalinn.  Tlie  geneialiy.alion  nf  t  he  [CSSS]  soi  ling 
re.sults,  nainelv.  l  ho  piribh.-in  <»f  s<n  ( ,\  items  i>i»  a  /■*-no<lc  liypercube  in  time 


when  /V  >  P,  is  open. 

-If  N  =  +  ,  then  i'P‘/‘  logP  =  (-)(P/rIr»g2  /V). 
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Using  tlie  result  of  Appendix  A,  we  can  see  that  if  Z  =  5", 

?sort(^.  Z/R)  =  ^sort(-‘>'')  S'/ R)  =  0(  —  log'  .S'). 

V 

If  Z  =  .V,  then 

Tsonrf^.  Z/R)  =  7’sort( N/R)  =  0{j  log'  N). 

'l  liis  bound  can  also  be  expres.sed  as 

0(^  log- 5’). 

.Since  r  <  log5,  the  hound  of  Section  4.3.2  simplifies  to 

0{^log^S+logP). 

Thus,  the  algorithm  for  the  general  case  takes  time 

0(R.\ogS+\ogP+-\og~S). 

r 

I 

4.5.3  Case  3:  P  <S  <  N 

Thcortfui  4.5.3  If  we  art  given  N  pairs  distributed  evenly  at  the  P  processors  of  a  hypercube,  then 
wt  can  perform  ,  P.  S)  in  lime 

OiR\ogS  +  -\og\S) 
r 

when  P  <  S  <  A^ 

Proof:  In  Section  4.4.1,  it  was  shown  that  if  /?  >  5  we  can  perform  0i'^(N ,  P,  S)  in  time 
0(R  tog  S)  when  P  <  S  <  N . 

In  Section  4.4.2,  it  weis  shown  that  \{  R  <  S,  we  can  perform  P, .S’)  in  time 

0(PlogP  +  TsoRT(A^P)) 

when  P  <  .S  <  .V.  From  Appendix  A,  we  know  that 

TsoRT(A^P)  =  0(-log-A) 
r 

if  those  constraints  resulting  from  the  [CS88]  algorithm,  mentioned  in  the  previous  section,  are 
saUsfied.  Now  since  R  <  ,S,  it  follows  that  N  <  SP  <  S'.  Thus, 

7sort(  A,  P)  =  0(  —  log'  ,S). 

r 

Hence,  the  time  bound  for  the  algorithm  of  Section  4.4.2  is 

0(— log-.S). 

V 

The  .ilgonihm  for  the  gener.nl  ca.se  takes  time 

0{  R  log  S  +  —  log"  .S'), 
r 


I 
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4.6  Performing  the  Beta  Operation  when  S 
is  Unknown 

In  Section  3.4.1,  we  considered  the  problem  of  creating  an  algoritlim,  Algorithm  B,  for  solving 
‘problem  P  not  given  output  size  .S’,'  using  as  a  subroutine,  an  algorithm,  AlgoiiLlnn  A,  for  solving 
‘problem  V  given  output  size  S’.’  In  Theorem  3.4.1,  we  showed  that  if  the  running  time  of  Algorithm 
A  wa-s  t(  A',  Q)  (where  Q  is  a  given  bound  on  the  output  size),  then  Algorithm  B  could  be  constructed 
with  a  running  time  that  was  Q{t(N,Q)).  We  then  applied  this  result  to  the  hypercube  and  MO'l 
implementations  of  N ,  S). 

Theorem  3.4.1  can  also  be  applied  to  tin’  various  implennmtalions  of  /li ’*‘(.V,  P,  .S')  presented  in 
this  chapter.  Let  us  modify  conditions  (/)-("’)  and  (a)-(h)  of  Section  .3.4.1  by  replacing  t(N,Q)  with 
t{N,  P,Q).  If  we  have  an  algorithm  that  satisfies  the  modifu'd  conditions  (7)-(;u)  and  an  a.ssocialed 
gue.ss  sequence  that  sati.sfies  the.  modified  conditions  (a)-(6).  then  Theorem  3.4.1  holds,  as  before, 
with  t(N,Q)  replaced  by  t{N,P,Q)  and  i{N,Q)  replaced  by  i{N.P,Q)-  Lemma  3.4.1  can  then  be 
modified  to  produce  Lemma  4.6.1.  below, 

Lfunuia  4.6.1  Let  t{N,P,Q)  =  c{G{N,P)+  F{i\’,t\Q))  when  c  is  a  cuiistant  and  F(.\\P.Q)  />■ 
(I  ftniciton  that  is  monotonicaUy  increasivy  in  Q.  The  yiu  ss  scijut  iicc 

=  1,  //i  =  iinji{x-|  F{N.  P,r)  =  (('2'  -  \)(!(N.  P))},  i  >  0 
satisfies  the  modified  conditions  (a)-(h)  with  rj  =  rs  =  2. 

riie  algorithms  of  Sections  4 '2,  4.3,  aiul  11  satisly  the  modified  coiuliiions  {i)-{ii')  and  the 
associated  running  times  are  of  th<“  form  described  in  f.emma  4.6.1.  It  follows,  therefore,  from  the 
modified  Theorem  3.4.1  and  Lemma  4.6.1,  that  the  running  times  of  these  algorithms  do  not  increa.se 
by  more  than  a  constant  factor  if  S  is  not  known  beforehand. 


Chapter  5 

The  Multi-Prefix  Operation 


5.1  Overview 

III  this  chapter,  we  shall  consider  the  efficient  implementation  of  another  primitive  parallel  operation 
of  the  Connection  Machine,  the  multi-prefix  operation.  In  Section  5.2,  we  first  deline  tin'  n ml  ti- prefix 
operation.  In  Section  5.3,  we. demonstrate  how  02  can  be  implemented  efficiendy  by  preserving  a 
record  of  the  paths  of  the  data  items  during  the  implementation  of  the  simpler  formulation  of  tlie 
beta  operation,  /Ii.'  In  Section  5.4,  we  then  show  that  by  saving  additional  information  about  the 
partial  computations  calculated  during  our  implementation  of  /?2,  we  can  iniplenienl  the  mult  i-prefix 
operation  efficiently  on  a  hypercube.  We  shall  show  that  our  implementation  of  the  mnlti-profix 
operation  can  be  composed  from  the  same  set  of  primitives  that  was  used  in  our  imiilementalion 
of  02,  without  increasing  the  use  of  any  of  the  primitives  by  more  than  a  constant  factor.  One 
consequence  of  this  result  is  that  the  two  implementations  will  require  the  .same  amount  of  lime. 
We  can  use  these  two  iinplementations  as  paradigms  for  implementing  tlie.se  two  primitives  on  other 
networks.  We  first  define  the  multi-prefix  operation. 

5.2  Definition 

The  multi-prefix  operation  is  similar  in  form  to  the  02  operation.  Let  be  the  jtli  pioci-ssor  in 
proc(^Bi^.  Let  lij  bo  the  ordered  list  of  v-values  from  the  first  j  pairs  in  £?,.  If  the  input  funclion  is 
F,  the  multi-prefix  operation  computes  »/»j  =  Flhj,  for  j  €  G,  1  <  j  <  |ZI,|  and  leaves  (/, //,j  )  in  p,j  . 

Example  5.2.1  Let  F  be  ‘-h’  and  N  be  5.  Let  the  {g,v)-pairs  in  the  ivpnl  be: 

/uof  1  ,■  (9,6)  proc  2;  (2,4)  proc3;(3,4)  proc  4,' (9, .5)  pToc  5.  (2, -5) 

Per-forvtiiig  IIk  mull i-pre fir  operation  on  these  pairs  produces  the  output: 

proc  \ :  {9,(i)  proc  2.- (2,4)  proc3.(3,4)  proc  4;  (9,  1 1 )  /iroc  5;  (2,  9) 

*  Fr>r  convenience,  wr  shall  refer  to  /?]  (.S‘)  and  P2{-^)  ^  0\  ami  /?2  in  this  chapter. 
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5.3  Implementation  of  l3-2 

5.3.1  Overview 

We  start  by  reviewing  the  Section  3.2  implementation  of  the  operation.  In  tlie  naive  approach  to 
implementing  ,  we  would  st  art  by  sorting  all  the  pairs  by  their  group  values.  We  could  then  easily 
reduce  each  grou])  to  a  single  repre.sentative  value  and  route  these  values  to  their  final  destinations. 
The  problem  with  this  approach  is  that  the  initial  sort  consumes  too  much  time. 

The  Section  3.2  implementation  of  0\  takes  advantage  of  the  fact  that  the  number  of  pairs 
decrea.ses  as  a  result  of  the  combination  that  occurs  as  the  output  is  calculated.  This  decrease  in  the 
amount  of  data  allows  the  combination  process  to  speed  up  as  the  algorithm  progresses.-  During  each 
step  of  the  implementation,  the  data  are  separated  into  subcubes  of  size  .S'’,  and  0i  is  performed  on 
the  pairs  in  each  subenbe  to  produce  5  new  pairs.  These  new  pairs  are  themselves  grouped  together 
in  subcubes  for  the  ne.vt  step.  We  can  view  the  implementation  as  creating  a  circuit,  each  gate  of 
which  is  an  ,S"’-input,  ,S-output  gate  that  performs  0i.  The  S  pairs  that  are  output  by  the  final  gate 
in  this  circuit  represent  the  result  of  performing  0i  on  the  fV-node  hypercube. 

In  this  seclion,  we  shall  show  how  02  cai'  b*-  implemented  with  the  same  efficiency  as  0i.  The 
implementation  of  02  that  we  present  combines  pairs  in  the  same  way,  and  creates  the  same  circuit 
as  that  used  in  the  implementation  of  0i.  In  this  case,  however,  the  circuit  will  be  saved  so  that  the 
pairs  produced  by  the  final  gate  can  be  routed  back  to  their  initial  sources.  Our  implementation  of 
,Tj  will  bo  composed  of  the  same  set  of  primitives  that  was  used  in  our  implementation  of  0i. 

We  shall  use  the  same  notation  as  in  Section  3.2.  It  is  assumed  Ixdow  that  S  is  given.  For 
convenience,  let  us  call  the  implementation  of  0i  presented  in  Section  3.2,  I>npkiii€n1ation-0i,  and 
the  impleiiK'iitatioiis  of  Tj  and  the  multi-prefi,\  operation  presented  below,  [mplemeii1aUon-02  and 
Jmpiciiitiilnlion-M P ,  respectively.  Recall  that  0  is  the  binary  string  of  s  I’s  (i.e.,  S  —  1),  and  <!>,  the 
binary  string  of  s  —  1  I’s,  followed  by  a  zero  (i.e.,  S  —  2). 

5.3.2  The  General  Step 

The  general  step  of  both  Impleinentaiion-01  and  ImplemeniaUon-0i  is  the  same,  except  that  at 
the  end  of  the  sort  and  compact  stages,  we  now  store  pairs  for  future  use.  More  specifically,  at 
the  end  of  these  two  stages,  each  processor  stores  the  source  address  of  the  pair  it  is  holding,  in 
a  processor  whose  address  is  Hamming  distance  one  away.  We  shall  show  later  that  our  choice  of 
storage  locations  ensures  that  the  circuit  can  be  stored  with  no  node  needing  more  than  a  constant 
amomil  of  storage. 

As  in  lv)plevienialion-0i,  in  every  step  of  Implemen1ation-02,  we  conceptually  break  the  hyper¬ 
cube  into  a  number  of  smaller  subciibes  and  perform  several  primitive  operations  in  each  .subcube  in 
parallel.  Each  subcube  corresponds  to  a  gate  in  the  circuit  being  stored  and  each  step  corresponds  to 
performing  the  gate  operations  on  a  single  level  of  the  circuit.  In  the  steps  of  Pha.se  3,  the  addresses 
in  the  stored  circuit  are  used  to  route  the  output  values  back  to  the  initial  .sources. 


^This  approach  is  similar  to  the  filtration  paradigm  of  (HMS84]. 
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5.3.3  Implementafion-^2  -  Phase  1 

111  this  phase,  we  create  the  first  level  of  t.lie  circuit.  We  conceptually  break  the  hypercube  into  a 
number  of  subcubes  of  size  .S'"*.  Each  subcube  correspontls  to  a  first  level  gate.  Each  gate  has  S* 
input  pairs  and  5  output  pairs.  Thus,  in  this  phase,  the  number  of  pairs  is  reduced  from  N  to  N/S^. 
The  output  pairs  from  this  phase  reside  in  N/S*  subcubes  of  size  S. 

Instead  of  keeping  just  a  (i^.uj-pair  at  the  processors,  each  processor  will  maintain  a  (g,v,p)- 
triple,  where  p  is  a  processor  number.  The  g-  and  r-  .slots  are  filled  with  the  values  from  the 
(gi,  ij)-pair.  In  every  step,  each  processor  puts  its  address  in  the  p-slot. 

By  convention,  fc  is  0  for  the  one  step  of  this  pliase;  thus,  the  hypercube  addresses  are  of  the 
form  uovqwoXo,  where  I'Uol  =  D  —  4,  |iio|  =  3,  jtuol  =  0,  and  )xo|  =  1.  The  nodes  in  a  given  subcube 
have  addresses  of  the  form  There  are,  therefore,  |«o|  fir-st  level  gates  in  the  corresponding 

circuit.  The  Uo-th  such  gate  heis  inputs  of  the  form  a-nd  outputs  of  the  form 

For  each  subcube,  ff,  we  perform  the  following  three  stages: 

Sort  Stage.  Using  odd-even  merge  sort  we  sort  the  triples  by  g-value.  Each  processor  then  stores 
its  triple  in  its  local  memory. 

Reduce  Stage.  We  next  invoke  Lemma  3.2.2  to  reduce  each  Bi(H)  to  a  single  triple. 

Compact  Stage.  As  in  ImpkmtnUUton-iix,  we  route  the  final  pairs  in  each  5‘*-node  subcube  to 
the  5-node  subcube  with  addresses  of  the  form  All  processors  n6t  holding  a  final 

triple  set  their  triples  to  the  null  value.  The  final  triples  represent  the  output  of  the  first  level 
gates. 

We  now  save  the  information  tliat,  during  Phase  3,  allows  us  to  route  the  output  values  back 
to  the  initial  sources.  In  what  follows,  let  us  call  a  processor  holding  a  non-null  triple,  a  live 
processor.  Let  us  call  the  S  processors  with  the  highest  addresses  in  H,  the  live  set  of  H. 
Every  live  processor  with  an  address  of  the  form  U\)0^Xo  now  stores  a  copy  of  its  triple  in  the 
local  memory  of  the  processor  that  is  Hamming  di.stance  one  away,  with  address  UQ9^<f>Xo.  Let 
us  use  the  shorthand  notation,  Is.-omp.  to  refer  to  a  live  set  produced  by  the  compact  stage. 
As  noted  above,  the  Iscomy's  have  addresses  of  the  form  Upd^Xp. 

5.3.4  Implementation- ^2  -  Phase  2 

In  this  phase,  we  complete  the  •^uilding  of  t.lie  circuit.  The  number  of  triples  is  reduced  to  S  by 
repeated  applications  of  the  general  step.  There  are  D  —  4  steps.  Step  k  (1  <  i  <  D  —  4)  is 
performed  as  shown  below. 

As  in  Phase  1,  we  shall  view  the  processor  addn^sses  a.s  lieing  of  the  form  UkVkWkXk  during  Step 
k,  where  |u*|  =  D  —  4  —  k,  |nr.|  =  3,  [ku-I  =  k,  and  |j;r.|  =  1.  At  the  start  of  Step  k,  the  processors  with 
addresses  of  the  form  i  are  live.  We  start  .Step  k  by  breaking  the  A'/5*^  proce.ssors 

whose  addresses  are  of  the  form  VkVkO^Xk  into  A'/.S'*"*’'*  subculves  of  nodes  each  of  the  form 
UkVkB'‘ Xk.  Note  that  with  this  choice  of  partitioning  the  hypercube,  each  subcube  contains  at  most 
live  processors.  Thus,  each  corresponding  gate  has  .S'-  or  fewer  non-empty  inputs  and  .9  outputs. 
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As  in  Phase  1,  there  are  \ui;\  gates  on  level  L-  in  the  corresponding  circuit.  The  tT-th  sncli  gate  has 
inputs  of  the  form  Ui-Vi,-6'‘Xk  and  outputs  of  the  form  UtO'‘''^^Xk. 

Next,  each  live  processor  in  one  of  these  subcubes  creates  a  new  triple  by  copying  the  (/-  and 
v-values  from  the  triple  it  received  in  the  previous  compact  stage  into  the  new  y-  and  e-slols,  and 
putting  its  address  in  the  ;>-slot.  For  each  5‘‘-node  subcube,  H,  of  the  form  described  above,  we 
perform  the  following  three  stages: 

Sort  Stage.  We  can  use  the  Nassinii-Sahni  sort  [NS82]  to  sort  the  at  most  S'-  (g.  i',  p)-triples  in 
each  subcube  by  3-value.  When  the  sorting  is  complete,  each  live  processor  with  address  of 
the  form  Uk-iO'‘'^-Xk-i  stores  its  triple  in  the  processor  with  acldre.ss  Uk-\0<f>0^' Xk-\- 

Reduce  Stage.  We  reduce  each  Bi{H)  to  a  single  triple,  using  the  reduce  stage  of  Imj>lemen1  a Uon-Si . 

Compact  Stage.  We  then  compact  the  final  triples  into  the  processors  with  t  he  highest-numbered 
addresses  in  H .  When  the  compaction  has  been  completed,  each  live  processor  with  address 
of  the  form  UkO'^'^^Xk  stores  a  copy  of  its  triple  in  the  processor  with  addre.ss  U k X k  ■ 
As  in  Phase  1,  the  copy  will  be  used  to  enable  us  to  route  the  output  values  back  to  the 
initial  sources  during  Phase  3.  Following  the  notation  of  Phase  1,  we  shall  refer  to  the  live 
sets  produced  by  the  compact  stage  as  Iscomp's.  As  noted  above,  the  /s,.„,„p’s  liave  addresses 
of  the  form  UkO'^'^^Xk- 

5.3.5  Implementation-f^2  -  Phase  3 

Each  step  of  Pha.ses  1  and  2  corresponds  to  performing  the  gate  operations  on  a  single  level  of  the 
circuit.  To  enable  us  to  “rever.se”  the  gate  operations  on  a  level,  we  stored  triples  twice  during 
each  step,  once  during  the  sort  stage,  and  once  during  the  compaction  stage.  The  p-values  of  the 
triples  stored  in  the  sort  stage  allow  us  to  undo  the  sort.  The  p-values  of  the  triples  stored  in  the 
compaction  stage  allow  us  to  reverse  the  compaction  and  undo  the  effects  of  the  associated  reduction 
stage. 

It  is  important  to  keep  in  mind  that  all  the  live  sets  retrieved  during  this  phase  wdl  be  rei.rieverl 
in  tlie  e.xact  reverse  order  in  which  they  were  stored;  e.g.,  we  start  Step  k  of  Pha.se  3  by  retrieving 
the  Isconip's  that  ^vere  stored  in  the  compaction  stage  of  Step  k  of  Phase  2.  The  retrieved  live  sets 
allow  us  to  route  the  output  values  back  down  the  circuit,  one  level  at  a  time.  As  in  the  previous 
phases,  we  shall  view  the  processor  addresses  as  being  composed  of  four  fields,  ukOkfi'k-i'k,  during 
Step  k.  There  are  D  —  ‘i  steps.  We  shall  count  down  from  D  —  4  to  0.  Step  k  {D  —  4  >  k  >  0)  is 
performed  as  shown  below. 

Un-compact  Stage.  Wc  start  Step  k  by  retrieving  the  ;>-values  of  the  Iscomp  '^  that  were  stored  in 
the  compaction  stage  of  Step  k  ofPha.se  2.  The  processor  with  address  replaces  the 

p-value  of  its  triple  with  the  p-value  from  the  triple  of  the  processor  with  address  Uk9-0O''' Xk- 
The  new  p-value  in  each  triple  is  the  address  of  this  triple  prior  to  the  compact  stage  of  the 
Step  k  of  Pha.se  2.  With  a  monotone  route,  we  can  .send  each  triple  to  its  pre-compaction 
location. 
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Uu-i‘educe  Stage.  Every  processor  that  received  a  triple  during  the  monotone  route  of  the  un- 
compact  stage  now  calls  itself  a  leader.  Using  two  distribution-from-leaders  operations,  each 
leader  can  now  distribute  its  (<;,  v,  p)-triple  to  the  proces-sors  in  its  i>roc(^Hi{  H  on  either  side. 

Un-sort  Stage.  Finally,  we  retrieve  the  live  sets  stored  in  the  sort  stage  of  Step  k  of  Phase  2.  Let 
us  call  these  sets  the  The  live  triples  now  replace  their  p- values  with  the  values  of 

the  retrieved  /s,ort’s.  We  can  then  reverse  the  sort  of  Step  k  of  Pha.se  2  by  sorting  tlie  triples 
by  p-value. 


The  last  step  reverses  the  single  step  of  Phase  1  and  leaves  (i,  j/j)  in  each  member  of  proc(^Bi{H)^ , 
as  desired. 

Note  that  the  addresses  of  the  live  sets  stored  during  Step  k  of  Pha.se  2  have  a  4>  in  the  (jb-t-2)nd 
least  significant  digit  of  their  addresses.  The  addresses  of  the  live  sets  stored  in  Step  j,  for  j  >  k, 
disagree  with  the  addresses  of  the  live  sets  stored  in  Step  k  in  the  {k  2)nd  digit.  The  addresses  of 
the  live  sets  stored  in  Step  j,  for  j  <  k,  disagree  with  the  addresses  of  the  live  sets  stored  in  Step 
k  in  the  (j  +  2)nd  digit.  It  is  clear,  therefore,  that  no  more  than  a  constant  number  of  triples  are 
stored  in  any  processor  in  the  course  of  the  implementation. 


5.3.6  Time  Analysis 

Phases  1  and  2 

Each  act  of  storing  (or  retrieving)  triples  takes  constant  time,  as  every  processor  stores  its  triple 
either  in  local  memory  or  in  the  local  memory  of  a  processor  wliose  address  is  Hamming  distance 
one  away.  Aside  from  operations  taking  constant  time,  the  running  times  of  these  two  phases  are 
identical  to  their  counterparts  in  Implementation-pi.  From  Section  3.2  we  know  that  these  two 
phases  can  be  performed  in  time 

0{logN  -f  TsortC^,  S')). 


Phase  3 

The  sort  used  in  the  un-sort  stage  of  Step  k  {k  >  0)  can  be  performed  with  the  Nassimi-Sahni 
algorithm  in  time  O(logS).  The  sort  used  in  the  un-sort  stage  of  Step  0  can  be  performed  in  time 
0{5sort(5', 5)).  The  remaining  two  stages  require  only  a  constant  number  of  distribution-from- 
leaders  operations  and  monotone  routes,  each  of  which  takes  time  O(log.S). 

Step  k  (k  >  0)  takes  time  U(log5)  and  there  are  0{\ogN/  logS)  steps.  Step  0  takes  time 

0(logS  -f-  Tsort(5,  S)). 

Tims,  Pha.se  3  can  be  performed  in  time 

0{\ogN  +  TsoKr{S,S)). 

The  overall  time  used  by  the  implementation  is,  therefore, 

0{\ogN  -b  Tsort('S',  .S’)). 
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5.4  Implementation  of  the  Multi-Prefix 
Operation 

5.4.1  Overview 

Iwplevieniation-M P  is  created  by  addiii}:;  a  number  of  simple  modifications  onto 
Irnplementation-p^ ■  Specifically,  there  are  four  important  modifications. 

•  We  first  perform  a  pre-processing  phase,  Pha.se  0,  that  allows  us  to  2»ssume  that  the  jj'-values 
of  the  initial  pairs  range  from  0  to  5'  —  1. 

•  In  Phase  1,  we  calculate  the  partial  prefixes  in  each  Bi(H)  instead  of  simply  reducing  each 
Bi(II)  to  a  .single  value. 

•  At  the  end  of  every  compact  stage  in  hnplemeniaiion-02,  we  routed  the  remaining  pairs  in 
each  5''-subcube,  //,  to  the  S  highest-address  processors  (the  live  set  of  H).  In  the  compact 
stages  of  Implemeniation-MP,  we  route  the  remaining  pair  from  group  i  (if  there  is  one)  to 
the  ith  processor  in  the  live  set. 

•  In  every  step  of  Pha,se  1  and  2,  there  is  an  additional  stage,  the  distribution  stage.  During 
each  distribution  stage,  we  shall  store  the  nodes  of  a  tree  of  (j,  x;,p)-triples.  These  nodes  will 
be  retrieved  in  Phase  3  to  produce  the  correct  values  of  j/jj . 

In  ,S<'ction  2.1,  we  stated  that  as  input  to  either  variant  of  the  beta  operation,  we  are  given  an 
as.sociative,  commutative  binary  function  /'  and  N  pairs,  (ga,vo),  ■  ■  ■  By  introducing 

minor  modifications  to  our  .Section  3.2  implementation  of  /?]  and  our  Section  5.3  implementation 
of  /?2,  we  can  create  implementations  that  do  not  require  F  to  be  commutative.  More  specifically, 
if  we  use  sorts  tliat  are  stable  and  preserve  the  ordering  during  the  simple  tree  operations  of  Step 
3  of  Lemma  3.2.2,  the  implementations  presented  for  Ri  and  /?2  will  work  correctly  even  if  the 
input  function,  F,  is  not  commutative.  .Similarly,  we  shall  show  that  for  the  implementation  of  the 
multi-prefix  operation  presented  below,  F  is  not  required  to  be  commutative. 


5.4.2  Implementation- MP  -  Phase  0 

Given  an  instance  of  the  multi-prefix  operation,  we  store  the  input  pairs  and  run  Implementation -R^ 
to  the  end  of  Phase  2.  At  this  point,  the  number  of  triples  has  been  reduced  to  S  and  the  circuit  has 
been  stored  for  use  in  Phase  3.  With  a  simple  prefix  operation,  each  triple  can  determine  its  rank 
(from  0  to  6'  -  1).  At  the  end  of  Pha.se  3,  each  triple  replaces  its  (/-value  with  its  rank  and  restores 
its  u-value.  We  may  now  assume  that  the  (/-values  of  the  initial  pairs  range  from  0  to  5  —  1. 

5.4.3  Implemenlntion-MP  -  Phase  1 

In  this  phase,  we  create  the  first  level  of  the  circuit.  As  in  Implementation- R2,  we  break  the  hypercube 
into  a  number  of . subcubes  and  reduce  the  number  of  triples  in  each  subcube  from  N  to  N/S^.  For 
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each  subcube,  //,  we  also  calculate  the  partial  prefixes  in  each  We  use  the  same  three  stages 

of  Phase  1  in  Impkinenlaiion-d-i  with  tlie  following  modifications.  A  fourth  stage  is  also  added 
below:  the  distribution  stage. 

Sort  Stage.  In  order  to  ensure  that  the  correct  partial  prefixes  ci.re  calculated  in  the  next  stage,  it 
is  necessary  that  the  sort  be  stable.  We  can  achieve  a  stable  sort  by  using  two  keys.  VVe  sort 
the  triples  by  3- value  with  an  odd-even  merge  sort;  however,  if  two  triples  have  equal  3-values, 
we  compare  Revalues.  Each  processor  then  stores  its  triple  in  its  local  memory. 

Reduce  Stage.  VVe  can  compute  the  partial  prefixes  for  each  Bi{H)  by  invoking  a  trivially  modified 
version  of  Eemnia  i.'l.'l.  To  start,  every  processor  stores  its  addre.ss  in  the  p-slot  of  its  triple. 
We  then  apply  the  first  two  steps  of  Lemma  3.2.2.  Upon  completion  of  the  second  step,  each 
processor  in  Cli;  contains,  at  most,  four  triples.  Using  simple  tree  operations,  we  can  now 
compute  the  partial  prefi.xes  for  each  Bi{H)  in  parallel,  taking  care  to  preserve  the  ordering 
of  the  triples.  VVe  store  the  resulting  values  in  the  u-slots  of  each  triple.  Using  the  addresses 
stored  at  the  beginning  of  this  stage,  we  can  route  the  triples  back  to  their  original  sources  by 
reversing  the  monotone  routes  of  Step  2  of  Lemma  3.2.2.  Each  proces.sor  then  stores  its  triple. 

Compact  Stage.  If  there  is  a  pair  in  H  from  group  <  after  the  reduction  stage,  we  route  it  to  the 
tth  procc'ssor  in  /7’s  live  set. 

Distribute  Stage.  When  the  reduction  stage  has  been  completed,  we  shall  have  calculated  the 
value  of  F  applied  to  the  Bi(Hys  in  each  separate  subcube,  H.  To  compute  the  j/,j ’s,  we  must 
combine  tlie  value  of  F  applied  to  the  /B,(/f)’s  in  each  subcube  with  the  values  of  F  applied  to 
the  corresponding  f?i(//)’s  in  all  subcubes  with  lower  addresses.  In  this  stage,  we  shall  create 
a  tree  of  (3,  v,  p)-triples  with  each  node  in  the  tree  representing  the  set  of  values  produced  by 
applying  F  to  the  Si(//)’s  in  a  number  of  i/’s.  During  Phase  3,  the  nodes  of  this  tree  will  be 
recalled  in  the  reverse  order  in  which  they  were  produced  in  order  to  form  the  pij  values. 

Recall  that  we  referred  to  the  live  sets  produced  by  the  compact  stage  as  Iscomp's.  At  the 
completion  of  the  compaction  stage  in  Implenientation-fln,  a  copy  of  each  Iscomp  with  address 
of  the  form  was  stored  in  the  processors  with  addresses  of  the  form  UpO^ipXo.  At  the 

beginning  of  the  distribution  stage,  each  processor  makes  a  copy  of  its  stored  /s„,„p  triple.  We 
shall  call  these  new  copies,  /-Srf,-,(’s,  by  analogy. 

The  distribution  stage  is  basically  a  prefix  operation  where  the  values  operated  on  are  .9-tuples. 
More  specifically,  when  we  complete  this  stage,  each  will  contain  the  S  or  fewer  values 

representing  the  resiilt.s  of  F  applied  to  the  triples  from  all  in  the  containing  block 

whose  addresses  are  strictly  lower.  Note  that  this  partial  prefix  operation  is  not  inclusive:  i.e., 
at  the  l•nd  of  the  stage,  the  new  value  of  lsdi,t  does  not  Include  its  old  value. 

Westart  In  grouping  thesiibcnbes  together  in  hhrk.sol S  subenbes  each.  Consider  a  particular 
block,  L.  To  start,  cnany  proces.sor  in  L  that  holds  a  triple  stores  its  address  in  the  p-slot.  We 
next  send  every  Bi{H]  to  a  rlifferent  .9-node  hypercube  in  L  as  follows.  If  there  is  a  triple 
from  group  /  in  the  jth  /.s,(,:,,,  it  is  sent  to  the  yth  position  in  the  ith  9-node  hypercube  in  L. 
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Note  that  the  triples  in  such  a  hypercube  are  ordered  by  their  original  source  addresses.  Tliose 
processors  that  did  not  receive  a  triple  as  a  result  of  the  last  operation  now  create  a  dummy 
triple  with  u-value  0.  Using  simple  tree  operations,  we  can  now  compute  the  iioii-inclu.sive 
partial  prefixes  in  each  5-node  subcube.  The  resulting  v-values  are  stored  in  tlte  e-slots  of 
the  associated  triples.  Finally,  we  route  all  triples  (including  the  dummy  triples)  in  a  given 
5-node  subcube  back  to  the  5  /srf,j<’s.  The  triple  in  the  jth  position  in  the  /th  such  subcube 
is  sent  to  the  I'th  position  in  the  jth  The  triples  computed  in  this  stage  are  stored  in 

local  memory. 

5.4.4  Impltmentation- MP  -  Phase  2 

In  this  phase,  the  number  of  triples  is  reduced  to  5^  by  repeated  application  of  the  general  step.® 
At  the  end  of  the  phase,  all  the  information  for  determining  the  j/jj’s  is  stored  iu  Ihe  tree  of  triples. 
There  are  D  —  5  steps.  The  first  three  stages  of  Step  k  {I  <  k  <  D  ~  b)  are  the  same  as  those  of 
Step  k  of  Phase  2  of  lmplementaiion-P-2,  with  two  modifications.  First,  in  the  monotone  route  of 
the  compaction  stage,  we  route  the  remaining  pair  from  group  i  (if  there  is  such  a  pair)  to  the  ?'th 
processor  in  H’s  live  set.  Secondly,  as  in  Phase  1,  after  the  compact  stage,  there  is  an  additional 
stage,  the  distribution  stage.  The  distribution  stage  of  Step  k  has  the  .same  form  as  in  Phase  1.  As 
before,  the  sort  of  the  sort  stage  must  be  stable;  we  achieve  a  stable  sort  by  using  the  (/-value  as  a 
primary  key  and  the  p- value  as  a  secondary  key.  The  Isdist's  have  addresses  of  tlio  form  UkO-<f)9^Xii. 

5.4.5  Implementation- MP  -  Phase  3 

During  Phaise  2,  we  stored  triples  three  separate  times  during  each  step.  As  in  IinpleiiiCDlaiion-p^t 
the  p-values  of  the  triples  stored  during  the  sort  and  compact  stages  were  used  to  create  a  circuit 
that  allows  us  to  route  the  output  results  back  to  the  appropriate  proces.sor.s.  In  addition,  we  also 
created  a  tree  of  triples  representing  values  of  F  applied  to  various  subcubes. 

More  concretely,  the  end  value  of  a  particular  i/,y  is  determined  as  follows.  At  the  beginning  of 
the  distribution  stage  of  Step  k  of  Phase  2,  the  with  address  Ui.  0‘^<f>9''' represents  F  applied 
to  the  f-values  from  the  subcube  of  proces.sors  with  addresses  of  the  form  Uk  Vk  Wk  .Yt . 

At  the  end  of  this  stage,  each  lsii,t  represents  all  the  processors  that  were  represented  by  those 
Isdtsi's  in  the  containing  block  with  lower  addresses.  The  value  of  yij  is  calculated  by  combining 
those  Isdist’s  that  represent  the  processors  with  addresses  lower  than  pij.  As  in  [vipleinentation-P2, 
all  the  live  sets  retrieved  during  this  phase  are  retrieved  in  the  reverse  order  that  they  W('re  stored. 
Note  that  the  retrieval  process  maintains  the  correct  ordering  among  the  triples.  There  are  Z?  —  4 
steps.  We  count  down  from  D  —  b  to  0. 

We  start  Step  k  (D  -  b  >  k  >  1)  by  retrieving  the  IsdisFs  that  were  stored  in  the  distribution 
stage  of  Step  k  of  Phase  2.  The  Isdiat  with  address  of  the  form  U kO- ^  is  retrieved  into  the 
tscomp  with  address  of  the  form  Uk9'‘+^.^  The  u- value  from  each  lsdi,t  triple  is  transferred  to  the 

^For  technical  reasons,  there  is  one  fewer  step  in  this  Fha.se  2  than  in  the  f-^hase  2  of  fmplrmpuiofion-f^'y- 
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v-slot  of  the  corresponding  /sco,,,,.  triple.  Afterwards,  tlie  reverse  routing  of  the  triples  of  tlu'  Income  s 
on  level  k  of  the  circuit  is  identical  in  form  to  Step  k  of  Pluvse  of  Imjilcmcntal ion-l3-2 . 

Finally,  we  repeat  these  four  stages  one  more  time  with  k  et|ual  to  0,  in  order  to  reverse  Phase  1. 
One  modification  is  introduced  in  the  last  step.  At  the  end  of  the  un-reduce  stage,  the  triples  stored 
uuriiig  the  reduce  stage  of  Phase  1  are  retri»*vet!  and  th-'  ii-vahie  of  each  retrieved  triple  is  comhiiK'd 
with  the  ii-value  of  the  triple  already  contained  in  the  proce.ssor.  The  following  un-sort  stage  leaves 
Dij  Pij  3®  desired,  and  completes  the  implementation. 

5.4.6  Time  Analysis 

In  Section  5.3,  we  presented  an  efficient  implementation  of  /I2  in  terms  of  a  number  of  simple 
primitives.  Aside  from  0(1)  operations,  four  modifications  are  applied  to  hnpleinental ton to 
create  Implemeniation-MP .  These  modifications  are  enumerated  in  Section  5.4.1.  We  now  show 
that  these  modifications  do  not  increase  the  u.se  of  any  of  the  .simple  primilivos  by  more  than  a 
constant  factor. 

The  first  modification  is  the  addition  of  a  pre-processing  pha.se,  Phase  0.  In  this  phase,  we  ensure 
that  the  jr-values  of  the  initial  pairs  range  from  0  to  .S-1.  This  phase  recpiires  us  to  perform  the 
three  phases  of  Implementation-iii  plus  a  single  tree  operation  on  S  items.  The  second  modification 
replaces  one  tree  operation  with  another  in  the  reduction  stage  ofPha.se  1  of  ImpUineniation Tlie 
third  modification  simply  changes  the  destination  addresses  in  the  monotone  route  of  the  compact 
stage.  The  fourth  modification,  the  addition  of  the  distribution  stage,  uses  only  a  constant  number 
of  the  primitives  already  required  by  the  sort  and  reduce  st  ages. 

5.5  Conclusion 

In  developing  Implementation- MP,  we  first  reviewed  the  implementation  of  from  Section  3.2. 
A  circuit  was  created  in  the  course  of  this  implementation.  In  Section  5.3,  we  showed  that  tliis 
circuit  could  be  used  to  create  an  implementation  for  jlo.  We  then  showed  in  Section  5.4  that  the 
implementation  of  02  could  be  modified  to  create  an  implementation  of  the  multi-prefix  operation 
without  increasing  the  complexity  of  the  computation.  One  consequence  of  the.se  results  is  that  the 
implementations  of  the  02  and  the  multi-prefix  operations  presented  in  this  chapter  can  be  used 
as  paradigms  for  implementing  these  two  primitives  on  other  networks.  Thus,  for  example,  we  can 
modify  the  Section  3.3  MOT  implementation  of  0\  to  obtain  MOT  implementations  of  02  and  tlie 
multi-prefix  operation  with  the  same  running  times. 

In  Implementation-02  and  [tnplemcnlaliov- M P .  we  assumed  that  S  was  known.  Nonetheless, 
from  the  result  of  Section  3.4,  it  follows  that  the  same  time  liounds  hold  even  if  S  is  not  known 
beforehand. 

Tlie  multi-prefix  operation  can  Ix'  generalize<|  by  making  !•'  dependent  on  the  //-value.  If  tlu'ie 
is  a  different  function,  Fg,  a,s.sociated  with  each  group,  the  above  analysis  remains  valid. 


Chapter  6 


Applications 

6.1  Simulating  Beta  Operations 
(with  Other  Beta  Operations) 

6.1.1  Overview 

In  Cliapici  1.  we  stated  that  a  primitive  operation  should  balance  efficiency  and  generality.  In  tliis 
section,  wo  clemonstraP?  this  halaiico  for  the  beta  operation  by  simulating  N,  S)  using  only 

applications  of  d(.s')  (for  S'  <  S).  'I’lie  generality  is  evidenced  by  the  fact  that  we  need  no  other 
primitives.  Tlie  elficiency  is  evidenced  by  the  fact  that  tiie  time  requirements  match  tliose  of  the 
the  most  efFuient  known  algorithm  for  performing  N,S).  We  shall  show  that  if  .S’'  = 

for  some  constant  c  >  1,  then  the  above  sinuilation  can  be  performed  on  an  iV-node  hyper''>ibe  with 
0(c-  +  \og  i\' /  ]ng  S’’ )  applications  of  /3(S').'  For  a  constant  c  >  1,  the  Section  3.2  algoritmn  takes 
as  long  in  the  ca.se  of  S  =  output  groups  as  it  does  in  the  case  of  S  =  IV  output  groups.  The 
simulation  of  this  section  shows  that  the  two  cases  are  “just  as  hard”. 

In  Section  ().1.2,  a  number  of  auxiliary  theorems  needed  for  our  result  are  established.  The 
algorithm  for  the  simulation  is  given  in  Section  6.1.3.  In  Section  6.1.4,  the  total  required  time  is 
calculated. 

6.1.2  Auxiliary  Theorems 

Below,  we  prcsi-nl  liv('  simple  auxiliary  theorems  needed  for  establishing  the  main  result  of  this 
section  (Theorem  6.1.6).  Let  log  S'  be  called  s'.  For  convenience,  we  shall  call  the  binary  string 
composed  of  s'  I  s  (i.e..  S'  —  1 ),  /t,  and  the  1  inary  string  composed  of  s'  —  1  I’s  (i.e.,  S'/'2  —  1),  7. 
As  before,  the  binary  st  ring  composed  of  s  I’s  (i.e.,  S  —  1)  will  be  called  9.  We  shall  assume  below 
that  there  is  never  more  than  one  pair  per  processor. 

The  first  theorem  demonstrates  that  sorting  can  be  performed  using  beta  operations. 

*  As  in  Sections  ^.S.2  anfl  ‘I.S..3.  we  shall  note  c  explicitly  in  the  hig-O  notation  although  it  is  a  constant. 
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Figure  6.1;  Composition  of  Hypercube  Addresses. 

Theorem  6.1.1  N  items  can  be  sorted  .stably  on  an  N -processor  hypercube  with  one  application  of 

A+(.v,/v,;v). 

Proof:  Let.  processor  pj  hold  item  kj  initially.  Each  processor  forms  a  (jr,r;)-pair  with  the 
j-vahie  set  to  (pj ,  Kj)  and  the  u-value  set  to  0.  A  given  p-value,  {pj  ,Kj},  is  less  than  another  ^-value, 
if  either  kj  <  kji  or  Kj  =  Kji  and  pj  <  pj,.  Clearly  no  two  ^-values  are  equal.  Thus,  the 
N  items  can  be  sorted  stably  with  one  application  of  /3i'*'(JV,  N,  N).  | 

If  the  input  (3,  u)-pairs  were  initially  sorted,  we  could  quickly  reduce  the  number  of  pairs  with 
repeated  applications  of  l3i(S').  Thi.s  concept  is  expres.sed  as  Theorem  6.1.2. 

Theorem  6.1.2  Suppose  we  are  given  an  N-node  hypercube  holding  N  (g,  v)-pairs  that  are  sorted 
by  g-value.  Then  every  Bi  can  be  reduced  to  a  single  pair  using  0(\ogN/\ogS')  applications  of 
Hi(S'). 

Proof:  Let  log^V  be  n  and  tj  be  [^5^1  •  reduction  of  the  Bi's  is  achieved  with  i)  steps. 
We  shall  view  the  processor  addresses  as  being  composed  of  four  fields,  UkVkWkXk,  where  k  is  the 
number  of  the  step  we  are  performing.  The  lengths  of  fields  Uk  and  Wk  are  functions  of  k.  The 
addresses  are  compo.sed  as  shown  in  Figure  6.1. 

In  each  step  of  the  algorithm  presented  below,  we  shall  conceptually  break  the  A^-node  hypercube 
into  a  number  of  smaller  subcubes  of  size  S' .  We  shall  reduce  the  number  of  pairs  by  performing 
(S' ,  S' ,  S' )  in  each  subcube  in  parallel.  We  .shall  distinguish  between  two  different  sets  of  pairs 
that  remain  after  the  reduction.  Recall  that  the  pairs  are  sorted  by  g-value.  Consider  a  subcube,  H. 
After  the  reduction,  only  the  first  and  last  pairs  contained  in  H  can  belong  to  groups  that  appear 
in  pairs  outside  of  H .  As  in  Section  4.4.2,  we  shall  call  these  two  pairs  reducible,  since  they  can 
be  combined  with  other  pairs  in  later  steps.-  We  shall  call  the  S'  -  2  or  fewer  pairs  that  appear 
between  the  reducible  pairs  irreducible. 

At  the  beginning  of  Step  k.  the  remaining  reducible  pairs  will  be  broken  into  subcubes  of  size  S' 
with  addresses  of  the  form  14-14  'y‘‘ A k-  Step  k  (0  <  fc  <  ?;  —  1)  is  performed  as  shown  below. 

Reduce.  We  perform  0i'*'(S' ,  S' .  S')  on  ‘•ach  ^'-node  subcube  with  addresses  of  the  form  1414  Xk. 

Compaction.  We  want  to  route  the  two  reducible  |>aii's  to  the  two  highest  addresses  in  H  and  tlie 
S'  -  2  or  fewer  irreducible  pairs  to  the  low^est  addre.'-'.'^e.s  in  //,  'flii.s  routing  can  be  performed 
sim|dy  with  four  /?i(5'')  operations.  The  highe.st-ad dress  prores.sor  stores  the  reducible  pair  it 
holds.  We  then  perform  Si  (S')  to  move  the  pairs  to  the  lowest-numbered  processors  in  H. 

^Tliere  will  be  only  one  reducible  pair  if  all  S'  pairs  in  //  come  from  (lie  same  group. 
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Figure  6.2:  Composition  of  lly|'erciil)e  Addresses. 


The  lovvest-address  processor,  wliile  remembering  its  true  ly-valuc,  sets  its  ^-value  to  infinity. 
Another  /?i~{5')  then  moves  the  irreducible  pairs  to  tlie  lowest  addresses  in  H  while  moving 
the  lower-numbered  reducible  pair  to  the  first  prore.ssor  after  the  irreducible  pairs.  Two  more 
fii(S'ys  suffice  to  route  the  lower- numbered  reducible  pair  to  tJie  .second  highest  location  in  H 
(first  move  the  pair  to  the  highest  location).  The  reducible  pairs  are  now  contained  in  locations 
Note  that  the  reducible  pairs  maintain  their  .sorted  order. 

After  T)  —  I  steps,  there  are  S'  or  fewer  remaining  reducible  pairs  in  the  S'-node  subcube  with 
addresses  of  the  form  1”"-*  We  perform  one  more  j3i'*'(S' ,  S' .  S')  on  this  subcube.  All 

the  pairs  are  now  irreducible.  .Since  no  two  irreducililo  jrairs  can  have  the  same  '/-value,  every  B, 
must  have  been  reduced  to  a  single  representative  pair. 

I 

We  next  present  two  theorems  concerning  simple  tree  operations  on  the  (</,  t')-p3ir.s  of  a  hyper¬ 
cube. 

Theorem  6.1.3  Suppose  tue  are  given  an  N -node  hypercubc  holding  N  or  fewer  {g,  v)-patrs.  Then 
we  can  perform  fdi*  {N ,  N ,  1)  ivHh  0{log  N/ logS')  ldi*{S'  ,S',l}  operations. 

Proof:  The  processor  addresses  will  be  viewed  as  being  composed  of  three  fields,  utvti-ut,  where  h 
is  the  number  of  the  step  we  are  performing.  The  addresses  are  composed  as  shown  in  Figure  6.2. 

Let  T)  be  Infs').  There  are  :/  step.s.  At  the  liegiiming  of  Step  k  (ft  <  k  <  rj  —  1).  the  remaining 
pairs  will  be  broken  into  subcubes  of  size  S'  witli  addresses  of  the  form  UtVtp''.  Bi^[S',S'.  1)  i.s 
performed  on  each  subcube.  After  tj  —  1  steps,  there  are  S'  or  fewer  remaining  pairs  in  the  5'-node 
subcube  with  addresses  of  the  form  V,,-i  I"'-'  .  We  perform  one  more  ‘''(S'.i'',  1)  on  this  subcubc 
to  reduce  the  remaining  pairs  to  a  single  value.  | 

Theorem  6.1.4  We  can  perform  broadcast  in  an  N -node  hypercube  using  only 
0( log  A/ log  S')  /?2 ( .S' , ,  1 )  operations. 

Proof:  The  proof  mirrors  the  form  of  the  proof  of  Theorem  6.1.3.  I 
The  previous  two  th<:orems  are  used  in  Theorem  6.1. -6  below. 


Theorem  6.1.5  Suppose  we  are  given  an  N'-node  hypercube  holding  N  items  in  an  N  node  .mb- 
ciibe.  Then  we  can  perform  SOKT{N ,  using  only  0(\og.N  flog  S')  Bi(S',S',  1)  and  f32(S',S',  1) 
operations. 
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Figure  6.3:  Composition  of  Hypcicube  Addresses. 

Proof:  We  shall  view  the  N'-processor  hypercube  as  an  jV  x  N  matrix,  yij,  with  Ihe  processors 
arrayed  in  order  of  decreasing  processor  number  (in  row-major  form).  Without  loss  of  generality, 
we  shall  assume  that  initially,  only  the  first  row  of  processors  contains  items,  bet  the  processor  in 
column  i,  pij,  hold  item  lij.  We  shall  act  as  if  all  the  items  are  different  by  using  t  he  metuod  of 
Theorem  6.1.1. 

We  start  by  invoking  Theorem  6.1.4  twice.  We  broadccist  the  contents  of  each  fust  row  processor, 
pij,  to  the  column  j.  Then  every  processor,  pjj,  broadcasts  its  contents  to  row  j.  Each  processor, 
Pij,  compares  the  items  received  in  the  two  broadcasts  and  puts  a  1  in  local  memory  if  the  value 
from  pij  is  greater  than  that  from  pyy.  The  sum  of  the  comparison  bits  in  a  column  is  then  the  rank 
of  the  item  from  pij.  The  column  sums  can  be  computed  by  invoking  Theorem  6.1.3  to  perform 
a  single  i3i'^{N,N,  1)  operation.  If  rj  is  the  rank  of  /cy,  then  we  can  route  i.-j  to  pi,.^  with  three 
more  broadcast  operations:  piy  broadcasts  to  column  j;  pjj  broadcasts  to  row  ;;  pjr^  broadca.sts  to 
column  ry . 

I 

6.1.3  Simulation  of  the  Beta  Operation 

The  auxiliary  theorems  of  Section  6.1.2  are  used  in  the  following  theorem. 

Theorem  6.1.6  Suppose  we  are  given  an  N -node  hypercuhe  holding  N  pairs  representing  S  different 
groups.  Let  S'  =  for  some  constant  c  >  1.  Then  the  application  of  Si'^ (N ,  N ,S)  can  he 
simulated  with  0(c-  -I-  logfV/logS')  applications  of  0i{S')  and  /?2(S'). 

Proof:  The  algorithm  below  is  closely  patterned  after  the  Section  3.2  algorithm  for  computing 
Ji^lN.N..S). 

Let  [u/s]  —  3  be  called  i].^  There  are  rj  steps.  We  shall  view  >.iie  processor  addresses  n.s  being 
composed  of  four  fields,  itkVkWkXk,  where  k  is  the  number  of  the  step  we  are  performing.  The  lengtiis 
of  fields  Uk  and  wi-  are  functions  of  k.  The  addresses  are  composed  as  shown  in  Figure  6.3. 

In  Step  k  of  this  algorithm,  we  shall  conceptually  break  the  A'-node  hypercube  into  a  number 
of  smaller  snbenbes  of  size  S'*  with  addresses  of  the  form  Uk  Vk  6^  A  k .  'Ihe  algorithm  is  divided  into 
two  pha.ses. 

Phaso  1.  riieri'  is  only  one  step  in  Phase  I.  By  convention,  it  wall  be  called  Step  ().  In  Sti'p  0,  tin' 
hypeiTnlx'  is  broken  into  .V/.S'^  subcubes  of  size  3“^  with  adriresses  of  I  he  form  /  The 

pairs  in  each  subenbe  can  be  sorted  with  O(c-)  sorts  of  size  0(3'}  by  in  voicing  Corollary  A. 3. 2 

If  .s’  >  ,V.  can  pcirorm  the  sinnilalion  V>y  applying  Pha.se  1  helnw  tn  tin-  entile  hyperriibe. 
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with  N  equal  to  S  and  f  equal  to  4.  These  sort-,  can  be  performed  with  O(c-)  applications 
of  iiy  invoking  Theorem  (i.I.l.  We  then  invoke  Theorem  6.1.2  with  N  f^qnal  to  5'*  to 

reduce  the  number  of  pairs  in  each  .'J'^-node  subcube  to  at  most  S,  using  0(s/s')  applications  of 
6i(S')A  With  one  adclilional  application  of  Corollary  A. 3. ‘2  and  Theorem  6.1.1,  the  remaining 
pairs  can  be  routed  to  tlu'  .S'  higliest-address  processors  in  their  subcube  with  a  sort. 

Phase  2.  Phase  2  is  composed  of  jj-  1  steps.  In  Step  k  (0  <  k  <  f?— 1),  the  liypercube  is  broken  into 
subenbes  of  size  .S''^  with  addresses  of  the  form  UkV^O’'' Note  that  each  subcube  contains 
S-  or  fewer  pairs  We  can  thus  invoke  Theorem  6.1.5  with  N  equal  to  to  sort  the  pairs 
with  0(s/s')  /?i(5',5'.l)  and  .S',  1)  operations.  By  Theorem  6.1.2,  the  number  of 

pairs  in  each  .?^-node  snbeube  can  then  be  reduced  to  S  with  0(s/s')  applications  of  /?i(S'). 
With  one  additional  application  of  Theorem  6.1.5,  the  remaining  pairs  can  be  routed  to  the  S 
highest.-addre.ss  processors  in  their  subcube.  In  Step  r/—  1,  the  above  actions  are  performed  on 
the  subcube  of  size  .5^  with  address  of  the  form  l""‘*''A',,_i.  Since  p  —  1  is  0{n/s)  and 
each  step  requires  O(sjs')  applications  of  and  fhiS'),  0(n/s')  applications  of  t3i{S') 

and  t)-j(S')  are  used  in  this  phase. 

I 

As  long  a,s  the  .sorting  subroutine.s  used  are  stable,  the  above  simulation  also  works  correctly  for 
the  version  of  N,S)  in  which  F  is  not  required  to  be  commutative. 

6.1.4  Time  Analysis 

Wehavp  shown  in  Section  6.1.3  that  N,S)  can  be  simulated  using  0(c~  ■+  n/s')  applications 

of  di(.S')  and  S'i{S').  In  this  section,  we  show  that  this  simulation  takes  the  same  time  as  the  most 
efheieni  known  implementation  of  the  Section  3.2  algorithm  for  0i'^{N,N,  S). 

Tliooioin  G.1.7  The  appttcaiion  of  N,S)  on  an  N -node  hypercube  holding  N  items  can  be 

sniiiildUd  in  time  0(log,N  +  log'.?)  using  only  0i{S')  and  02(S')  operations. 

Proof:  Consider  the  algorithm  of  Tlieorein  6.1.6. 

Phase  1.  There  is  only  one  step  in  Phase  1  of  Theorem  6. 1.6.  This  step  begins  and  ends  by  invoking 
both  Corollary  A. 3. 2  and  Theorem  61.1  to  sort  the  pairs  in  each  S^'-node  subcube  with  O(c-) 
applicaf  ions  of  0i  (S' ,S',  S').  The  invocation  of  Theorem  6.1.2  with  N  equal  to  S'*  that  occurs 
between  tliese  sorts  uses  only  0(c)  applications  of  0i(S' ,  S' ,  S').  From  Section  3.2,  we  know 
that  each  application  of  0\(S' ,S' ,S')  takes  time  7sort(5'', 5"').  Thus,  the  total  time  required 
for  this  plia.se  is  0(c-  ■  Tsort(.?'.  •?'))  W'e  know  that  tliere  is  an  upper  bound  (possibly  loose) 
lor  sorting  on  the  hypercube  model  that  we  have  been  using,  namely: 

Tsort(W,7V)  =0(log^V). 


^No(e  that  s/s'  =:  0(c). 


6.2.  EXPLOITING  LOCALITY 


47 


Thus, 

rr-  ■  TsoR-r(.‘?'.5'')  =  0(c^-  ■  \/c^  ■  log'  S). 
Tlio  overall  time  for  this  phase  of  the  simulation  is  thus  0(log'  S). 


Phase  2.  Phase  2  is  composed  of  tj-  1  steps.  Step  k  begins  and  ends  by  invoking  Theorem  6.1.5  to 
sort  the  pairs  in  each  5‘^-node  subcul)e  with  0(s/s')  Pi{S',S' ,  1)  and  /?2(•S^•S',  1)  operations. 
Both  MS\S\  1)  and  p'AS\S\l)  take  time  O(s'). 

By  Theorem  6.1.2,  the  number  of  pairs  in  each  S^-node  subcube  can  be  reduced  to  S  with 
0(.s/s')  applications  of  iJi(S').  AiiS')  is  performed  in  the  subcubes  with  addresses  Uk^k 
By  Theorem  4.5.1,  /?i(S'')  can  be  performed  in  time  O(s')  in  each  cube,  since  the  number  of 
proces.sors  is  the  square  of  the  number  of  items. 

Since  -  1  is  0(n/s)  and  each  step  requires  0(s/s')  operations  that  each  require  time  O(s'), 
the  time  required  for  this  phase  is  0{n). 


The  total  time  required  by  the  algorithm  of  Theorem  6.1.6  is 

C>(log  ;V  +  log'  S) 
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6.2  Exploiting  Locality 

6.2.1  Overview 

In  this  section,  we  present  a  simple  application  of  the  generalized  beta  operations  presented  in 
Chapter  4.  In  particular,  using  Theorem  4.5.1,  we  can  show  that  if  the  (5,t))-pairs  of  the  input  are 
“bunched  together,”  the  time  required  for  the  hypercube  implementation  of  l3i'^{N,N,S)  can  be 
reduced  to  log  IV. 

6.2.2  A  More  Efficient  Implementation  of  /?i 

We  demonstrate  the  following  result: 

Theorem  6.2.1  Suppo.'ie  we  are  given  an  instance  of  the  beta  operation  to  perform  on  the  hypen  ube, 
X,S).  If,  for  a  constan  t  c  >  0,  all  .subcubes  of  size  contain  pairs  from  no  more  than  x 
different  groups,  then  we  can  perform  the  beta  operation  in  time  0{logN). 

Proof:  We  can  achieve  this  result  by  adding  a  simple  modification  to  the  Section  .3.2  algorithm 
for  implementing  [N ,  N .  .S)  on  a  hypercube.  The  modified  algorithm  has  three  phases. 

Phase  1.  We  start  by  breaking  tlie  hypercube  into  subcubes  of  size  2'/'°*'®  and  perform¬ 

ing  the  /li'*'  operation  on  each  of  these  subcubes.  From  Section  3.2,  we  know  that  this  plia.se 
can  be  performed  in  time  O(log.S). 
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Phase  2.  We  next  group  the  subcubes  of  Phase  1  together  to  form  larger  subcubes  of  size  S'*.  We 
know  that  after  Phase  1,  eacli  cube  of  size  S*  has  at  most  3*0+-:  remaining  i>airs.  We  can  now 
invoke  Theorem  4.5.1  to  perform  the  /3i*'{S*0+‘^S*,S)  operation  on  each  subcube  in  lime 
0(log5). 

Phase  3.  At  the  end  of  Phase  2,  tlie  condition  of  the  remaining  pairs  is  identical  to  that  which 
pertained  at  the  end  of  Phase  1  in  the  Section  3.2  algorithm.  To  complete  the  A,  S) 

operation,  we  simply  apply  Phase  2  of  the  Section  3.2  algorithm.  This  last  phase  takes  time 
O(logA). 
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Chapter  7 

Summary 


In  the  preceding  chapters,  we  have  examined  the  beta  operation  proposed  by  Hillis  ([Hil85]).  In 
Chapter  2,  we  presented  two  variant  formulations  of  the  beta  operation,  0\  and  0-,- 

In  Chapter  3,  we  studied  the  problem  of  implementing  liie  beta  operation  efficiently  on  several 
multiprocessor  networks,  namely,  the  hypercube,  mesli.  and  mesh-of-trees  grapits.  The  hypercube 
implementation  was  presented  first  and  served  as  a  paradigm  for  the  other  implementations.  We 
showed  that  all  three  implementations  could  be  performed  in  time 

(?(log  N  +  7soaT(-S‘.  .S')), 

where  TsoKriN,P)  is  the  minimum  time  required  to  sort  N  items  on  a  given  P  processor  network. 
The  identity  of  the  network  in  question  is  understood  by  context. 

We  next  proved  that  any  when-  and  where-determinate  circuit  that  can  perform  beta  operations 
for  any  S  requires 

AT-  =  U(NS\ogN). 

This  lower  bound  is  close  to  the  upper  bound 

AT^  -  0(N  log-  N(S  -I-  log-  N)) 
realized  in  the  mesh-of-trees  implementation. 

In  Chapter  4,  we  considered  the  problem  of  implementing  the  beta  operation  efficiently  when 
the  relationship  among  N,  P,  and  S  was  allowed  to  be  general.  We  denoted  the  three  possible 
relationships  as: 

Case  1:  S  <  N  <P 

Case  2:  S  <  P<N 

Case  3:  P  <  S  <  N 

We  presented  implementations  of  the  beta  operation  for  these  three  cases  that  satisfied  the 
following  upper  bounds. 
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Case  1; 

Case  2: 


(log 


O  logP  + 


log- 5  \ 

log  p/nJ 


ftlog5+ 


Case  3: 

Ill  Chapter  5,  we  re-cxaniinecl  tlie  hypercube  implementation  of  /^i  from  Chapter  3  and  studied 
the  hypercube  implementation  of  the  other  variant  of  the  beta  operation  discu&sed  in  Chapter  2, 
/?2-  We  showed  that  by  creating  a  circuit  that  stores  information  about  the  partial  computations 
that  arise  during  the  hypercube  implementation  of  /?i,  we  could  implement  /?2  with  no  increase 
in  complexity.  We  next  presented  a  new  primitive  that  generalizes  the  beta  operation,  the  multi¬ 
prefix  operation.  We  Liion  demonstrated  how  our  implementation  of  ^2  could  be  augmented  to 
provide  an  implementation  of  the  multi-prefix  operation,  again  with  no  increrise  in  complexitj'.  The 
implementations  of  the  02  and  multi-prefix  operations  serve  as  paradigms  for  implementing  these 
two  primitives  on  other  networks. 

In  Chapter  6,  we  presented  some  applications  of  the  beta  operation.  We  demonstrated  both  the 
efficiency  and  the  generality  of  the  beta  operation  by  simulating  complex  beta  operations  using  only 
simpler  beta  operations.  Lei  S'  =  5^^'  for  some  constant  c  >  1.  We  showed  that  the  application 
of  0\'^{N,N,S)  could  be  simulated  with  0(c* -f  log W/ log 5')  applications  of  0i{S')  and  02[S'). 
This  simulation  entailed  only  a  constant  factor  increase  in  the  required  time.  Lastly,  we  presented  a 
sample  application  of  one  of  the  re.sults  established  in  Chapter  4.  We  showed  that  if  the  input  pairs 
are  “bunched  together,”  the  time  required  for  the  hypercube  implementation  of  0i'^{N,N,S)  can 
1)6  reduced. 


Chapter  8 


Open  Questions 


In  Chapter  3,  we  studied  the  problem  of  implementing  the  beta  operation  on  several  multiprocessor 
networks.  Given  these  implementations,  we  may  ask  more  broadly  how  the  structure  of  a  network 
relates  to  its  ability  to  execute  this  primitive  operation  (or  any  other).  We  may  ask,  for  example,  if 
there  is  a  good  way  of  characterizing  the  networks  that  cein  sort  quickly.  If  the  networks  are  described 
in  a  group-theoretic  way,  is  there  some  way  of  establishing  broad  classes  of  networks  whose  group 
properties  ensure  the  ability  to  perform  certain  primitives  efficiently? 

In  Chapter  4,  we  considered  the  efficient  implementation  of  P,  S)  for  all  possible  relation¬ 

ships  among  S,  N ,  and  P.  In  Sections  4.2,  4.3,  and  4.4,  the  time  requirements  for  these  implemen¬ 
tations  were  all  expressed  in  terms  of  the  SORT(N,P)  operation.  In  Section  4.5,  we  provided  actual 
time  bouiuls  by  using  the  hypercube  implementation  of  the  [CS88]  sorting  algorithm.  The  [CS88] 
result  states  that  if  IV  =  pi+i/®  for  some  constant  c,  then  Tsokt{N,P)  is  0{c  P^^‘\ogP)  on  a 
shuffle-exchange  graph  and  0(c-  P'/'logP)  on  a  hypercube.  In  Appendix  A,  we  show  that  a  slight 
modification  to  the  [CS88]  hypercube  implementation  reduces  the  required  time  to  0(c  P*/‘logP). 
Recently,  [Pla88]  has  demonstrated  that  on  a  stronger  hypercube  model,  SORT(N,  P)  can  be  per¬ 
formed  in  time  0(1:  log  P),  when  PlogP  =  IV*+^/*.  It  remains  to  be  shown  whether  optimal  sorting 
can  be  achieved  on  the  stronger  hypercube  model  for  all  cases  where  N  >  P.  Also,  the  question  is 
still  open  whether  the  [Pla88]  results  can  be  carried  over  to  the  weaker  model  of  the  hypercube  that 
was  introduced  in  Chapter  2. 

Our  implementation  of  N,S)  takes  time  0(log  A  -I-  log"  S),  as  opposed  to  0(log^  N)  for 

the  straightforward  implementation.  This  time  savings  is  the  result  of  our  using  the  first  phase  to 
reduce  the  amount  of  data  and  free  up  enough  processors  to  perform  the  second  phase  quickly.  Is 
this  paradigm  of  general  applicability?  [HMS84j  show  that  filtration  is  of  u,se  in  several  particular 
cases.  Do  algorithms  that  benefit  from  the  use  of  some  type  of  filtration  siiare  a  common  element? 
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Appendix  A 

Sorting  on  a  Hypercube 


A.l  Overview 

In  this  appendix,  we  address  the  problem  of  sorting  N  items  on  a  P-processor  hypercube.  In 
Section  A. 2,  we  consider  the  problem  of  sorting  when  N  =  P'+i/'  for  some  constant  c.  [CS88] 
demonstrate  a  network-independent  algorithm,  Cubesort,  that  performs  tliis  sort.  They  also  demon¬ 
strate  how  this  sort  can  be  implemented  in  time  0{c  log  P)  on  a  shuffle-exchange  graph  and 
time  0(c*  P^/'  log  P)  on  a  hypercube.  We  shall  modify  the  [CS88]  hypercube  implementation  of 
Cubesori  so  that  it  also  takes  time  0{c  P^/'logP)  (Theorem  A. 2. 3); 

In  Section  A. 3,  we  consider  the  problem  of  sorting  on  a  hypercube  wlien  N  =  P.  There  are  many 
sorting  algorithms  that  can  be  implemented  efficiently  on  a  hypercube  whose  running  times  match 
the  best  known  upper  bounds  (e.g.,  [Bat68]).  For  the  purposes  of  Section  6.1,  however,  we  need  an 
algorithm  that  sorts  N  items  on  an  A-processor  hypercube  using  only  sorts  of  size  polynomially  less 
than  N. 


A. 2  Performing  SORT(N,P)  on  a  Hypercube 

A. 2.1  Notation 

We  begin  by  reviewing  the  [CS88]  notation.  [CS88]  choose  two  variables,  M  and  D,  satisfying  three 
conditions; 

•  =  N 

•  M>  {D-l)iD-2) 

•  D>Z 

In  what  follows,  the  addresses  of  the  N  items  will  be  viewed  as  being  composed  of  D  base- 
M  digits.  We  shall  view  these  D  base-Af  digit  item  addres,ses  as  being  composed  of  four  fields, 
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Figure  A. 1:  Composit  ion  of  Hypercube  Addresses. 

utVkWkXk-  The  lengths  of  fields  ut  and  Xk  are  functions  of  k.  The  fields  are  composed  as  shown 
in  Figure  A.l.  We  shall  view  the  D  —  2  base-Af  digit  processor  addresses  as  being  composed  of  the 
three  fields,  Uk,  Vk,  and  Xk- 

[CS88]  define  D  partitions  of  the  items,  Pk,  for  1  <  fc  <  D.  Each  partition,  Pk,  will  divide  the 
items  into  blocks  of  size  M-.  In  the  special  case  where  k  =  1,  Pi  divides  the  items  into  ^  blocks. 
The  items  in  one  block  of  a  particular  partition,  Pk,  have  addresses  of  the  form  Ui-VkW^Xk-'  [CS88] 
employ  two  operations  wherein  all  the  blocks  of  partition  Pj  are  sorted  in  parallel;  Sort-Ascending  (J) 
and  SoTi-Mixed(k,j).  In  Sort -Ascending {j),  each  block  of  partition  Pj  is  sorted  in  ascending  or¬ 
der.  In  Sort-Mixed{k,  j),  each  block  of  partition  Pj  is  sorted  in  ascending  order  if  14  is  even  and 
descending  order  if  14  is  odd. 

A. 2. 2  Cubesort 

[CS88]  present  Cubesort  (below)  and  prove  it  correct. 

Cubesort (D) 
integer  D; 

{ 

ii  D  =  3  then 

Limit _Dirty_Cubes  (D); 

Sort_Mixed  (D-1,D-1); 

Merge_Dirty_Cubes  (D,D); 

} 

else  /*  D  >  3  */ 

{ 

Limit_Dirty_Cubes  (D) ; 

Limit _Dirty_Cubes  (D); 

Cubesort  (D-1); 

Merge_Dirty_Cubes  (D,D); 

} 

> 


’The  item.s  of  P\  have  adrliesses  tliat  vary  only  in  (he  last  base-iVf  digit. 
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Limit_Dirty_Cubes  (D) 
integer  D; 

{ 

if  D  >  2  then 

Limit_Dirty_Cubes  (D-1) 
Sort_Ascending  (D) ; 

} 

Merge_Dirty_Cubes  (D,T) 
integer  D,T; 

Sort_Mixed  (D,T); 
if  T  >  2  then 

Merge_Dirty_Cubes  (D,T-2) 

} 


Consider  tlic  t  wo  subroutines  of  Cubesori,  LimitJDirty.Cubes  (5)  and 
Merge.DirtyX^ib(?s  (S’).  We  can  unroll  LimitJDirty .Cubes  {S)  to  get 

Limit _Dirty_Cubes  (D) 
integer  D; 

For  j  =  2  to  D 

Sort_Ascending  (j); 

} 

Merge-Dirty .fUibes  (S')  can  al.so  be  unrolled,  to  get 

Merge_Dirty_Cubes  (D,T) 
integer  D,T; 

For  j  =  0  to  Ceiling(T/2)-l 
Sort .nixed  (D,T-2j); 

> 


Tlius,  Cube  sort  is  compo.sed  of  a  list  of  Sorl-Ascendtng{j)  and  Sort.Mir€d(k,  j)  operations, 
bet  us  call  an  instance  of  either  Sori.Ascending{j)  or  Sort.Mixed{k,j),  local-sort  j  (or  just  local-sort 
if  it  is  not  important  which  particular  partition  is  involved).  For  the  purposes  of  implementation,  it 
is  important  to  determine  the  difference  in  partition  number  (i.e.,  subscripts)  between  consecutive 
local-sort's  during  an  application  of  Algorithm  Cubesort.  Let  us  call  the  event  wlierein  the  subscript 
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of  one  local-sort  diffei's  from  that  of  the  previous  local-sort  by  e,  8^.  That  is,  the  event  where 
local-sort j+e  follows  local-sort. j  for  some  j  is  6f  In  the  two  implementations  of  Cubesort  below, 
(Theorems  A. 2. 2  and  A. 2. 3),  M  is  chosen  so  that  each  processor  holds  one  block  when  the  items  are 
evenly  distributed.  We  shall  say  Pj  is  earren/  if  its  P  associated  blocks  are  stored  one  per  processor.^ 
Let  A'  be  an  operator  that  applies  to  partitions.  Define  A'  to  be  such  that  if  Pj  is  current  and 
A‘(Pj)  is  performed,  then  Pj+e  is  current  afterwards. 

It  is  important  to  note  the  relationship  between  the  event  8  and  the  operation  A.  When  the 
event  8f  occurs  in  Algorithm  Cubesort  it  means  that  a  local-sort  follows  a  local-sort  j  for  some 
j.  In  order  to  perform  local-sort efficiently  in  the  implementation  of  Cubesort,  partition  Pj+c 
must  be  current  instead  of  Pj.  This  is  the  exact  result  of  applying  A‘  to  Pj.  Lastly,  we  shall 
call  a  permutation  of  P  elements,  (one  per  processor),  on  our  P-processor  hypercube,  a  simple 
permutation. 

The  following  two  theorems  paraphrase  the  [CS88]  results.  The  first  theorem  is  network  inde¬ 
pendent: 

Theorem  A. 2.1  [CS88]  Let  two  variables,  M  and  D,  be  chosen  to  satisfy  three  conditions: 

•  =  N 

•  M  >  (D  -  1)(D  -  2) 

•  D  >  3 

Then  N  items  can  be  sorted  with  0{D~)  local-sorts  by  Algorithm  Cubesort.  During  Algorithm  Cube- 
sort,  events  of  the  form  /i±i  occur  O(D^)  times  and  events  of  the  form  6±e  (for  2  <  e  <  D)  occur 
0{D)  times. 


Theorem  A. 2. 2  [CS88J  We  are  given  N  items  distributed  evenly  at  the  processors  of  a  P-node 
hypercube,  where  N  =  pi+i/'  Jor  a  constant  c.  Let  M  be  P^G^  and  D  be  2(c-|-l).  If  P  >  (4c^  -f  c)"' 
and  c  >  L,  Cubesort  can  be  implemented  on  the  hypercube  using  only  O(D^)  local  sorts,  O(D^)  A^  ’s, 
and  O(D-)  simple  permutations.  The  time  required  for  the  implementation  is  0(c^  P^^'^  log  P). 

A. 2. 3  A  More  Efficient  Hypercube  Implementation 

We  improve  upon  the  time  requirements  of  Theorem  A. 2. 2  by  performing  A'  more  efficiently.  In 
Tlieorem  A. 2. 2,  [CS88]  perform  A'  in  time  0(P'/'logP).  We  start  by  introducing  our  implemen¬ 
tation  of  A‘.  We  shall  perform  A*  in  time  0(1  fc  P^P  logP). 

^P\  is  current  whenever  is  current.  Tims,  in  what  follows  we  shall  not  consider  the  special  case  of  P\ . 
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Implementing  Efficiently 

The  two  lemmas  below  provide  us  with  algorithms  for  performing  A^  in  the  implementation  of 
Cubesort  presented  in  Theorem  A. 2. 3. 

Lemma  A. 2.1  Suppose  we  are  given  an  M  processor  hypercuhe  with  processors,  pt,  for  k  in  the 
range  0  to  M  —  I,  and  A/^  items.  labc  (0  <  a.b.c  <  A/).  (It  is  assumed  that  M  is  a  power  of 
2.)  Initially,  each  processor  pu  holds  the  Af-  items  hsf  (0  <  b,c  <  M).  We  can  permute  the 
items  in  time  0(M^\ogM)  so  that  after  ike  permutation,  processor  pk  holds  the  items  la/,/.- 
(0<a,b<M). 

Proof:  The  permutation  can  be  achieved  with  the  following  simple  divide-and-conquer  algo¬ 
rithm.  We  divide  the  processors  into  two  groups,  each  of  size  M/2.  Each  processor  in  the  higher- 
numbered  group  sends  M- /'2  values  to  its  counterpart  in  the  lower-numbered  group.  More  specif¬ 
ically,  each  processor  pt,  k  >  M/2,  sends  the  M‘^/2  items,  htc  (0  <  6  <  Af,  0  <  c  <  M/2) 
to  processor  Pk-M/2-  Symmetrically,  each  processor  pt,  k  <  M/2,  sends  the  Af^/2  items,  Ikie 
(0  <  b  <  M,  M/2  <  c  <  M)  to  processor  Pi-+.\f/2. 

Now  consider  the  lower-numbered  .M/2  of  the  processors.  Let  us  divide  the  M^  items  that  each 
processor  pk  contains  into  4  equal-sized  groups: 

•  hbe,  0<b<  M/2,0  <  c  <  M/2 

•  Ikbc,  M/'2  <  b  <  M,0  <c<  M/2 

•  4+M6c.  0  <  6  <  M/2,0  <  c  <  M/2 

•  4+M4c,  M/2  <  6  <  A/,0  <  c  <  M/2 

The  effect  of  the  permutation  of  Lemma  A.2.1  on  the  lower-numbered  M/2  of  the  processors  can 
be  achieved  with  four  recursive  applications  of  the  permutation  to  these  four  groups.  These  recursive 
permutations  can  be  applied  to  the  higher-numbered  processors  in  parallel.  If  T[M)  is  the  time 
required  to  perform  the  permutation  on  M  processors  and  AT®  items,  then  T[M)  <  ^  +  4T{~). 
Thus,  T(M)  is  0{M^-  log  M).  | 

In  Lemma  A.2.2  below,  a  method  is  presented  for  performing  A^  efficiently.  In  the  application 
of  Lemma  A.2.2,  the  proces.sors  are  grouped  together  in  units  of  size  M,  and  the  permutation  of 
Lemma  A.2.1  is  applied  to  the  items  contained  in  the  processors  in  each  given  unit.  We  follow  the 
lead  of  [CS88]  in  choosing  M  so  that  M-P  =  N . 

Lemma  A.2.2  Lei  N  Hems  he  dtsinhuted  m  a  P -processor  hypercube  so  that  a  given  partition  Pk 
{2  <  k  <  D  -  is  current.  Furthermore,  let  each  block  of  M-  items  with  addresses  of  the  form 
UkVkWkXk  be  contained  in  the  processor  with  address  of  the  form  UkVkXk-  In  time  0(M^\ogM), 
the  items  can  be  permuted  so  that  Pt  +  i  is  the  current  partition  and  each  block  of  M-  items  ivith 
addresses  of  the  form.  (4+i  Vi  +i  is  contained  in  the  processor  with  address  of  the  form 

Uk+\Vk+\Xk+i. 
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Proof: 

Initially,  the  blocks  of  partition  Pk  are  contained  in  the  processors  so  (  hat  each  block  of  M- 
items  with  addresses  of  the  form  UtVkWkXt  is  contained  in  the  proci'ssor  with  address  of  the  fortn 
UkVkXk-  Let  us  form  units  from  the  M  processors  with  addresses  of  the  form  UkVkXk-  Consider 
the  addresses  of  the  M^  items  in  a  particular  unit.  Call  the  left  and  rij!;h(.  posil  ions  in  field  Wk,  Wk^ 
and  Wk^-  Let  a, 6,  and  c  of  Lemma  A. 2.1  correspond  to  14,  and  114^4  respectively.  That  is, 
let  item  lau  of  Lemma  A. 2.1  correspond  to  the  item  with  address  UkobcXk  for  particular  values  of 
Uk  and  Xk-  We  can  then  invoke  Lemma  A.2.1  for  the  M  processors  and  M'*  items  of  each  unit  in 
parallel.  After  invoking  Lemma  A.2.1  the  processor  with  address  will  hold  the  M-  items 

with  addresses  of  the  form  UkW'kVkXk- 

By  definition,  for  any  address,  |ujt|  =  I'Uit+il  +  and  +  =  |m.|  +  |.rt|.  Let  us  break  the 

string  Uk  into  two  parts,  Uk+i  and  l4+i.  and  break  the  string  A4+i  into  two  parts,  14  and  Xk- 
Thus,  processor  address  UkVkXk  is  equivalent  to  fft+i  14+i  A4+1  and  the  item  addresses  of  the  form 
UkWkVkNk  are  also  of  the  form  Cft+iVt+iWit+iAt+i.  It  follows  that  after  invoking  Lemma  A.2.1, 
items  with  addresses  of  the  form  Cf/b+il4+i  144+1  are  held  by  the  processors  with  addresses  of 
the  form  t4+i^4+iA4+i,  as  was  to  be  shown. 

The  time  required  by  the  single  invocation  of  Lemma  A.2.1  is  0(M  -  log  A/). 

I 

The  following  corollary  provides  a  method  for  performing  A"'  efficiendy. 

Corollary  A.2.1  Lei  N  Hems  be  distributed  »n  a  P  processor  hypermlx  so  that  a  given  pariition 
Pk  (3  <  (t  <  £))  is  current.  Furthermore,  lei  each  block  of  M-  Hems  with  addresses  of  the  form 
Uk  Vk  Wk  Xk  be  contained  in  the  processor  with  address  of  the  form  UkVkNk-  The  items  can  be 
permuted  in  time  0{M^\ogM)  so  that  Pk-i  is  the  current  partition  and  each  block  of  M^  Hems 
with  addresses  of  the  form  17*_i  V4-iVV4_]  A4-i  is  contained  in  the  processor  with  address  of  the 
form  Uk-iVk-iXk-i- 

Proof:  The  proof  mirrors  the  proof  of  Lemma  A. 2. 2.  | 

The  New  Hypercube  Implementation 

With  the  aid  of  Lemma  A. 2. 2,  and  Corollary  A.2.1,  we  can  now  present  Theorem  A. 2. 3  that  demon¬ 
strates  the  more  efficient  hypercube  implementation  of  Algorithm  Ciibesorl. 

Theorem  A. 2. 3  We  are  given  N  Hems  distributed  evenly  at  the  processors  of  a  P-node  hypercube, 
where  N  =  +  for  a  constant,  c.  The  time  required  to  implement  (Uihesort  on  this  hypercube  is 

0(c  P^U  log  P)  when  P  >  {4c^  -f  and  r  >  i. 

Proof:  [CS88]  prove  Theorem  A. 2. 2  by  providing  a  hypercuhe  iniplemeiil ation  of  Cubesort.  The 
implementation  g'ven  below  follows  the  proof  of  Theorem  A. 2. 2  given  in  [CS88],  with  only  minor 
modifications. 
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Recall  that  Algorithm  Cuhesori  of  Theorem  A. 2.1  requires  0(D~)  local-sort'a.  local-soi  l  is 
peiTormecl  most  efficiently  if  each  block  is  stored  in  the  local  memory  of  one  of  the  processors,  i.e., 
if  Pk  is  current.  Towards  this  end,  we  follow  [CS88]  in  choosing  M  to  be  Pi  and  D  to  be  2(c -f  1). 
This  choice  allows  the  blocks  of  partition  Pk  to  be  contained  in  the  processors,  with  every  processor 
having  N/P  items.  The  conditions  of  Theorem  A. 2.1  are  .satisfied  when  P  >  (4c^  +  and  c  >  ~. 

We  can  now  invoke  Theorem  A. 2.1.  It  is  clear  from  above  that  if  partition  Pk  is  current,  we 
can  perform  local-sort  k  by  applying  a  standard  sequential  .sort  at  each  processor  in  parallel.  This 
operation  takes  time  O(M^logM)  and  occurs  O(D^)  times.  It  remains  to  demonstrate  that  we  can 
permute  the  items  so  that  Pk  is  current  when  local-sort  k  is  performed.  Initially,  P2  is  the  current 
partition.  We  know  from  Theorem  A. 2.1  that  during  Algorithm  Cubesort,  events  of  the  form 
occur  0{D‘^)  times  and  events  of  the  form  6±e  (for  <  e  <  D)  occur  0(D)  times.  For  each  event 
Sg,  we  must  perform  A'  in  our  implementation.  By  Lemma  A.2.2  and  Corollary  A. 2.1,  and  A“* 

can  be  performed  on  our  partitions  in  time  0(M^  log  M).  A'  can  be  performed  by  applying  A^, 
e  times.  Thus,  overall,  A^  and  A”^  must  be  performed  0(D“)  times.  These  A*  operations  can 
be  performed  in  time  0(D^M- log  A/).  [Ben65]  shows  that  a  fixed  permutation  of  P  elements  (one 
per  processor)  on  our  P  proces.sor  hypercube  can  be  performed  in  time  0{\ogP).  Thus,  the  O(D^) 
simple  permutations  take  time  0(D'^  log  P). 

Summing  these  three  time  bounds,  we  see  that  the  total  time  required  is 

0(0- log  M)  =  0  (c  P^''-  log  P) 


I 


A. 3  Performing  S0RT(N,N)  on  a  Hypercube 

.Sorting  is  an  important  sub-routine  in  our  .simulation  of  the  beta  operation  in  Section  6.1.  The 
following  theorem  addresses  the  problem  of  sorting  N  items  on  an  A-processor  hypercube  using  a 
number  of  sorts  of  size  less  than  N. 

Tlioorein  A. 3.1  N  items  can  be  sorted  or  an  N -processor  hyperenbe  with  O(c^)  parallel  sorts  of 
size  Q(N^P)  for  every  constant  c  >  1. 

Proof: 

Let  M  equal  and  let  D  equal  2c -P  1.  The  conditions  of  Theorem  A. 2.1  are  satisfied  if 

A  >  (4c"  —  2c)^'^+*.  Thus,  A  items  can  be  sorted  with  0(c^)  sorts  of  size  0(A'/'). 

It  remains  to  show  that  with  the  above  parameters,  Cubesort  can  be  implemented  on  an  A- 
proces.sor  hypercube.  This  result  can  be  shown  directly  from  Section  A.2.2.  Recall  that  Cubesort  is 
compo.sed  of  a  list  of  Sori.Ascendm(j(j)  and  SorlMued(k,  j)  operations.  Both  Sort_Ascending  ( j) 
and  Sort.Mixed  (l',i)  can  be  implemented  by  simply  performing  a  parallel  sort  in  the  hypercubes 
with  addresses  of  the  form  UjVjWjXj.  No  data  movement  is  needed.  | 

The  following  are  corollaries  of  Theorem  A. 3.1. 


A.3.  PERFORMING  SORT (N,N)  ON  A  HYPERCUBE 
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Corollary  A. 3.1  For  every  covstani  e  >  1,  Tsort(N‘,  N‘)  is  OiTsoKriN,  N)). 

Corollary  A. 3. 2  For  every  pair  of  constants  e,c  >  1,  Tsokt(N‘ ,  N^)  can  be  performed  on  an 
-processor  hypercube  with  0([ec)-)  sorts  of  size  0{N^/^). 
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